Sep 08 16:49:05.593860 ip-10-0-132-198 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 16:49:05.593868 ip-10-0-132-198 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 16:49:05.593875 ip-10-0-132-198 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 16:49:05.594092 ip-10-0-132-198 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 16:49:15.735974 ip-10-0-132-198 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 08 16:49:15.735993 ip-10-0-132-198 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bc5a0d2d6caf4a25be4f11bb8907c3de -- Sep 08 16:51:08.745595 ip-10-0-132-198 systemd[1]: Starting Kubernetes Kubelet... Sep 08 16:51:09.125093 ip-10-0-132-198 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:51:09.125093 ip-10-0-132-198 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 16:51:09.125093 ip-10-0-132-198 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:51:09.125093 ip-10-0-132-198 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 16:51:09.125093 ip-10-0-132-198 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 16:51:09.127139 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.127044 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 16:51:09.132446 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132425 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.132446 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132443 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.132446 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132448 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.132446 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132452 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132455 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132458 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132461 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132464 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132466 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132469 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132472 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132475 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132477 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132480 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132483 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132486 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132488 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132491 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132493 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132496 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132498 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132501 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132503 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.132612 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132506 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132508 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132512 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132514 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132517 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132520 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132524 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132526 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132529 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132531 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132534 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132536 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132539 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132542 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132545 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132548 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132551 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132554 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132557 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132559 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.133086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132562 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132564 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132567 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132570 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132572 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132575 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132577 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132582 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132584 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132587 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132590 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132592 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132595 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132597 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132600 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132603 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132605 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132608 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132611 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132613 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.133584 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132616 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132619 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132622 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132625 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132627 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132630 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132633 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132636 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132639 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132641 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132644 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132646 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132649 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132652 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132655 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132660 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132665 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132668 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132672 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.134070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132675 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132678 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132681 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132685 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132688 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132691 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132694 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132697 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132699 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132702 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132704 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132707 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132710 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132712 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132715 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132717 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.132721 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133420 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133428 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133432 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.134585 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133436 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133439 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133442 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133445 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133447 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133450 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133453 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133455 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133458 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133460 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133463 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133466 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133468 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133471 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133474 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133477 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133480 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133482 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133485 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133488 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.135070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133491 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133493 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133496 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133498 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133501 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133504 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133506 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133508 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133511 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133514 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133517 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133520 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133523 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133526 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133529 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133531 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133534 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133537 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133539 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133542 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.135583 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133544 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133548 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133550 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133553 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133555 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133558 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133560 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133563 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133566 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133569 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133571 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133574 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133578 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133581 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133584 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133587 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133590 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133593 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133596 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.136088 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133598 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133601 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133603 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133607 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133610 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133612 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133615 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133618 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133620 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133623 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133625 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133628 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133631 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133633 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133638 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133642 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133645 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133647 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133650 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.136586 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133653 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133656 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133659 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133662 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133664 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133667 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133670 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133672 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133675 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133677 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133680 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133683 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133687 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133690 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133693 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133695 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133698 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.133701 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134080 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134085 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.137068 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134088 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134091 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134093 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134096 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134099 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134101 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134104 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134107 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134109 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134112 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134114 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134117 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134119 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134122 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134125 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134128 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134130 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134132 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134136 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134138 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.137589 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134141 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134143 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134146 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134149 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134151 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134154 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134156 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134159 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134161 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134166 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134169 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134171 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134174 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134176 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134179 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134181 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134184 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134187 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134189 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134192 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.138086 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134195 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134197 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134199 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134203 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134207 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134210 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134213 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134217 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134220 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134222 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134225 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134228 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134231 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134233 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134236 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134238 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134241 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134244 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134246 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134249 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.138576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134252 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134256 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134259 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134262 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134264 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134267 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134269 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134272 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134275 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134277 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134280 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134284 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134287 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134290 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134293 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134296 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134299 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134302 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134305 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134307 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.139070 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134310 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134313 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134315 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134333 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134337 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134339 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134342 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134345 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134347 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134350 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134352 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134355 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134357 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134362 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134364 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134367 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.134371 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134440 2556 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134474 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134480 2556 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 16:51:09.139594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134484 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134488 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134491 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134496 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134499 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134503 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134506 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134509 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134512 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134515 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134518 2556 flags.go:64] FLAG: --cgroup-root="" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134521 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134524 2556 flags.go:64] FLAG: --client-ca-file="" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134528 2556 flags.go:64] FLAG: --cloud-provider="external" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134531 2556 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134536 2556 flags.go:64] FLAG: --cluster-domain="" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134539 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134542 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134545 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134548 2556 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134552 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134554 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134558 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 16:51:09.140095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134561 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134564 2556 flags.go:64] FLAG: --contention-profiling="false" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134568 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134570 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134574 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134576 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134580 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134583 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134586 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134589 2556 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134592 2556 flags.go:64] FLAG: --enable-server="true" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134595 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134599 2556 flags.go:64] FLAG: --event-burst="100" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134602 2556 flags.go:64] FLAG: --event-qps="50" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134605 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134608 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134611 2556 flags.go:64] FLAG: --eviction-hard="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134614 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134617 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134620 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134623 2556 flags.go:64] FLAG: --eviction-soft="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134626 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134629 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134632 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134637 2556 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 16:51:09.140666 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134639 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134642 2556 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134645 2556 flags.go:64] FLAG: --feature-gates="" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134649 2556 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134652 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134655 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134658 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134661 2556 flags.go:64] FLAG: --healthz-port="10248" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134664 2556 flags.go:64] FLAG: --help="false" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134668 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134671 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134674 2556 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134676 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134680 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134683 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134685 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134688 2556 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134691 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134694 2556 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134696 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134699 2556 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134702 2556 flags.go:64] FLAG: --kube-reserved="" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134705 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134707 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 16:51:09.141279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134710 2556 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134713 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134716 2556 flags.go:64] FLAG: --lock-file="" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134718 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134721 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134724 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134729 2556 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134734 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134737 2556 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134739 2556 flags.go:64] FLAG: --logging-format="text" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134742 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134745 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134748 2556 flags.go:64] FLAG: --manifest-url="" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134751 2556 flags.go:64] FLAG: --manifest-url-header="" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134755 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134758 2556 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134762 2556 flags.go:64] FLAG: --max-pods="110" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134766 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134769 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134772 2556 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134775 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134778 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134781 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134784 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134791 2556 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 16:51:09.141869 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134794 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134797 2556 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134800 2556 flags.go:64] FLAG: --pod-cidr="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134803 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134808 2556 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134811 2556 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134814 2556 flags.go:64] FLAG: --pods-per-core="0" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134817 2556 flags.go:64] FLAG: --port="10250" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134820 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134823 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00b0584666696f7db" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134826 2556 flags.go:64] FLAG: --qos-reserved="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134828 2556 flags.go:64] FLAG: --read-only-port="10255" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134831 2556 flags.go:64] FLAG: --register-node="true" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134834 2556 flags.go:64] FLAG: --register-with-taints="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134839 2556 flags.go:64] FLAG: --registry-burst="10" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134843 2556 flags.go:64] FLAG: --registry-qps="5" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134846 2556 flags.go:64] FLAG: --reserved-cpus="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134848 2556 flags.go:64] FLAG: --reserved-memory="" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134852 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134855 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134858 2556 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134861 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134864 2556 flags.go:64] FLAG: --runonce="false" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134867 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134870 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 16:51:09.142592 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134877 2556 flags.go:64] FLAG: --seccomp-default="false" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134879 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134882 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134885 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134888 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134891 2556 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134894 2556 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134897 2556 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134900 2556 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134903 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134906 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134909 2556 flags.go:64] FLAG: --system-cgroups="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134912 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134918 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134920 2556 flags.go:64] FLAG: --tls-cert-file="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134923 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134928 2556 flags.go:64] FLAG: --tls-min-version="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134930 2556 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134933 2556 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134935 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134938 2556 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134943 2556 flags.go:64] FLAG: --v="2" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134946 2556 flags.go:64] FLAG: --version="false" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134951 2556 flags.go:64] FLAG: --vmodule="" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134959 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 16:51:09.143237 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.134963 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135073 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135077 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135081 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135083 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135086 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135089 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135092 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135096 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135098 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135101 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135104 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135106 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135109 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135111 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135114 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135117 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135120 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135122 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135125 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.143875 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135129 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135133 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135135 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135138 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135140 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135143 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135145 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135148 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135152 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135156 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135159 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135163 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135165 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135168 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135171 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135174 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135177 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135180 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135183 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135187 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.144395 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135190 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135193 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135196 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135198 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135201 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135204 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135207 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135209 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135212 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135214 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135217 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135219 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135222 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135225 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135227 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135230 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135232 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135235 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135239 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.144885 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135242 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135246 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135249 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135252 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135255 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135258 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135260 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135263 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135265 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135268 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135271 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135274 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135277 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135280 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135282 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135285 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135288 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135290 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135293 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135295 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.145354 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135298 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135300 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135303 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135305 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135308 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135311 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135313 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135316 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135331 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135334 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135337 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135339 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135342 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135346 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135349 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135352 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135354 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135357 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135360 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.145845 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135363 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.135365 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.136049 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.143130 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.143146 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.144141 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.144149 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144212 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144217 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144220 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144224 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144227 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144230 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144233 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144237 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.146300 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144240 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144243 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144245 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144249 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144253 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144256 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144259 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144261 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144264 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144267 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144269 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144272 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144275 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144278 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144280 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144283 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144286 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144288 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144291 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.146740 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144294 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144296 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144300 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144303 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144306 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144308 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144311 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144314 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144317 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144333 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144336 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144339 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144341 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144344 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144347 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144349 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144352 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144355 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144358 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144360 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.147205 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144363 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144365 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144368 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144385 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144390 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144393 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144396 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144400 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144403 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144406 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144410 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144413 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144417 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144420 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144424 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144428 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144430 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144433 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144436 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144438 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.147699 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144441 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144444 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144447 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144449 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144452 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144454 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144457 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144460 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144463 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144465 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144468 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144472 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144475 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144478 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144480 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144483 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144486 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144489 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144492 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.148189 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144495 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144497 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144500 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144503 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144506 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144509 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144511 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144514 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144517 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144526 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144528 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144531 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144534 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.144539 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144684 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 16:51:09.148664 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144688 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144691 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144694 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144696 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144699 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144701 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144704 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144706 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144709 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144711 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144714 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144716 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144719 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144722 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144724 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144727 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144729 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144733 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144737 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 16:51:09.149025 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144740 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144743 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144746 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144748 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144751 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144754 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144757 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144759 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144767 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144770 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144773 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144775 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144778 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144780 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144782 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144785 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144788 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144790 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144793 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144795 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 16:51:09.149493 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144798 2556 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144801 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144803 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144806 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144808 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144811 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144813 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144816 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144818 2556 feature_gate.go:328] unrecognized feature gate: Example Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144821 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144823 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144826 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144828 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144831 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144833 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144835 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144838 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144840 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144843 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144846 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 16:51:09.149975 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144848 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144857 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144860 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144862 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144865 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144867 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144870 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144872 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144875 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144877 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144880 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144882 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144885 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144887 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144890 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144892 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144895 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144898 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144901 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144904 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 16:51:09.150478 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144906 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144909 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144913 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144916 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144919 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144922 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144924 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144927 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144929 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144933 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144936 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144939 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144942 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144944 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144954 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144956 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144959 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144962 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 16:51:09.150972 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:09.144964 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.144969 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.145083 2556 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.147302 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.148360 2556 server.go:1019] "Starting client certificate rotation" Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.148458 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.148490 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 16:51:09.151525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.150731 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:51:09.152208 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.152168 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 16:51:09.163886 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.163867 2556 log.go:25] "Validated CRI v1 runtime API" Sep 08 16:51:09.167659 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.167644 2556 log.go:25] "Validated CRI v1 image API" Sep 08 16:51:09.169708 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.169695 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 16:51:09.172582 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.172564 2556 fs.go:135] Filesystem UUIDs: map[431dff2b-cdae-4836-a3b7-f38259f1b9d8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b82b534c-3355-4e60-a59d-cf579c946797:/dev/nvme0n1p4] Sep 08 16:51:09.172641 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.172582 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 16:51:09.176810 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.176785 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:51:09.178648 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.178544 2556 manager.go:216] Machine: {Timestamp:2026-09-08 16:51:09.177296242 +0000 UTC m=+0.322066793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104012 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e52f7c193bf88a6bead5f359c540d SystemUUID:ec2e52f7-c193-bf88-a6be-ad5f359c540d BootID:bc5a0d2d-6caf-4a25-be4f-11bb8907c3de Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:72:0e:00:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:72:0e:00:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:d1:3b:ab:a9:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 16:51:09.178648 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.178646 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 16:51:09.178750 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.178722 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 16:51:09.181715 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.181691 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 16:51:09.181847 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.181719 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 16:51:09.182904 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.182894 2556 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 16:51:09.182940 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.182907 2556 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 16:51:09.182940 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.182914 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:51:09.183766 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.183755 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 16:51:09.184545 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.184535 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 16:51:09.188705 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188694 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:51:09.188814 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188805 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 16:51:09.188843 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188836 2556 kubelet.go:486] "Attempting to sync node with API server" Sep 08 16:51:09.188875 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188846 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 16:51:09.188875 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188863 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 16:51:09.188875 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188872 2556 kubelet.go:398] "Adding apiserver pod source" Sep 08 16:51:09.188979 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.188880 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 16:51:09.189257 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.189244 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:51:09.189297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.189269 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 16:51:09.191456 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.191438 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 16:51:09.193732 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.193718 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 16:51:09.193793 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.193747 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 16:51:09.194921 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194909 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 16:51:09.194968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194926 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 16:51:09.194968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194932 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 16:51:09.194968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194949 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 16:51:09.194968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194962 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 16:51:09.194968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194968 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194981 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194987 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194992 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.194998 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.195013 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 16:51:09.195092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.195019 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 16:51:09.196304 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.196291 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 16:51:09.196304 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.196301 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 16:51:09.198921 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.198890 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 16:51:09.198996 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.198927 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 16:51:09.199270 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.199252 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 16:51:09.199915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.199903 2556 server.go:1268] "Started kubelet" Sep 08 16:51:09.200010 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.199988 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 16:51:09.200162 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.199989 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 16:51:09.200222 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.200206 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 16:51:09.200746 ip-10-0-132-198 systemd[1]: Started Kubernetes Kubelet. Sep 08 16:51:09.204451 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.204127 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 16:51:09.205365 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.205223 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6t56" Sep 08 16:51:09.205586 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.205556 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 16:51:09.206375 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.206359 2556 server.go:310] "Adding debug handlers to kubelet server" Sep 08 16:51:09.208360 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.208343 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6t56" Sep 08 16:51:09.211005 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.210985 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 16:51:09.211103 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.210072 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-198.ec2.internal.18d366957b3b88c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-198.ec2.internal,UID:ip-10-0-132-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-198.ec2.internal,},FirstTimestamp:2026-09-08 16:51:09.199878343 +0000 UTC m=+0.344648895,LastTimestamp:2026-09-08 16:51:09.199878343 +0000 UTC m=+0.344648895,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-198.ec2.internal,}" Sep 08 16:51:09.211564 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.211546 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 16:51:09.211694 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.211678 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.212093 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212077 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 16:51:09.212093 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212080 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 16:51:09.212214 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212107 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 16:51:09.212214 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212169 2556 reconstruct.go:97] "Volume reconstruction finished" Sep 08 16:51:09.212214 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212176 2556 reconciler.go:29] "Reconciler: start to sync state" Sep 08 16:51:09.212299 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212247 2556 factory.go:55] Registering systemd factory Sep 08 16:51:09.212299 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212277 2556 factory.go:223] Registration of the systemd container factory successfully Sep 08 16:51:09.212483 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212473 2556 factory.go:153] Registering CRI-O factory Sep 08 16:51:09.212483 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212484 2556 factory.go:223] Registration of the crio container factory successfully Sep 08 16:51:09.212578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212522 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 16:51:09.212578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212538 2556 factory.go:103] Registering Raw factory Sep 08 16:51:09.212578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.212548 2556 manager.go:1196] Started watching for new ooms in manager Sep 08 16:51:09.213738 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.213721 2556 manager.go:349] Starting recovery of all containers Sep 08 16:51:09.218126 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.218107 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:09.221532 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.221508 2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-198.ec2.internal\" not found" node="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.224856 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.224720 2556 manager.go:354] Recovery completed Sep 08 16:51:09.227626 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.227605 2556 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 08 16:51:09.230788 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.230776 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.232683 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.232663 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.232771 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.232698 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.232771 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.232713 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.233253 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.233241 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 16:51:09.233253 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.233252 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 16:51:09.233387 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.233267 2556 state_mem.go:36] "Initialized new in-memory state store" Sep 08 16:51:09.233387 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.233247 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.234014 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.234001 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.234108 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.234026 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.234108 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.234046 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.235159 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.235146 2556 policy_none.go:49] "None policy: Start" Sep 08 16:51:09.235210 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.235163 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 16:51:09.235210 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.235173 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 16:51:09.237107 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.237096 2556 policy_none.go:47] "Start" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.270844 2556 manager.go:337] "Starting Device Plugin manager" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.270873 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.270881 2556 server.go:87] "Starting device plugin registration server" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.271163 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.271174 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.271269 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.271364 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.271373 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.271845 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 16:51:09.287791 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.271891 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.334812 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.334783 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 16:51:09.335886 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.335873 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 16:51:09.335956 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.335913 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 16:51:09.335956 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.335933 2556 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 16:51:09.336044 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.336003 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 16:51:09.338357 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.338340 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:09.372216 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.372197 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.372941 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.372928 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.373003 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.372954 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.373003 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.372964 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.373003 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.372985 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.381000 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.380945 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.381000 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.380964 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-198.ec2.internal\": node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.394818 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.394805 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.436574 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.436551 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal"] Sep 08 16:51:09.436663 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.436634 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.437454 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.437436 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.437529 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.437468 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.437529 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.437480 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.439790 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.439778 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.439912 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.439897 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.439947 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.439929 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.440473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440458 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.440523 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440483 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.440523 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440494 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.440594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440549 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.440594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440570 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.440594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.440588 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.442716 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.442702 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.442771 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.442727 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 16:51:09.443366 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.443351 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientMemory" Sep 08 16:51:09.443441 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.443377 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 16:51:09.443441 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.443394 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeHasSufficientPID" Sep 08 16:51:09.467835 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.467820 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-198.ec2.internal\" not found" node="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.471278 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.471264 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-198.ec2.internal\" not found" node="ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.495722 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.495701 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.596035 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.595995 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.613277 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.613253 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.613384 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.613285 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.613384 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.613309 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56dc1b1e4db5094fc00daf3962293d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-198.ec2.internal\" (UID: \"56dc1b1e4db5094fc00daf3962293d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.696702 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.696623 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.713891 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.713868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.713985 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.713901 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.713985 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.713925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56dc1b1e4db5094fc00daf3962293d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-198.ec2.internal\" (UID: \"56dc1b1e4db5094fc00daf3962293d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.714075 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.713987 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56dc1b1e4db5094fc00daf3962293d7e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-198.ec2.internal\" (UID: \"56dc1b1e4db5094fc00daf3962293d7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.714075 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.713991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.714075 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.714005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed925c216bdbc5803365eb4ce2b6827-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal\" (UID: \"fed925c216bdbc5803365eb4ce2b6827\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.770040 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.770005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.775689 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:09.775002 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:09.796772 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.796747 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.897272 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.897231 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:09.997754 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:09.997690 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-198.ec2.internal\" not found" Sep 08 16:51:10.025754 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.025736 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:10.112198 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.112166 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" Sep 08 16:51:10.125396 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.125379 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:51:10.127255 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.127241 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" Sep 08 16:51:10.136354 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.136338 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 16:51:10.148723 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.148702 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 16:51:10.148871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.148851 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:51:10.148927 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.148890 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:51:10.148927 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.148890 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 16:51:10.190391 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.190365 2556 apiserver.go:52] "Watching apiserver" Sep 08 16:51:10.207893 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.207870 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 16:51:10.209883 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.209863 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-6c526","openshift-multus/multus-additional-cni-plugins-tdpks","openshift-multus/network-metrics-daemon-j74bn","openshift-network-diagnostics/network-check-target-sdrwc","openshift-ovn-kubernetes/ovnkube-node-dsbwj","kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc","openshift-cluster-node-tuning-operator/tuned-t6r27","openshift-image-registry/node-ca-22bfg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal","openshift-network-operator/iptables-alerter-wh9tv","kube-system/global-pull-secret-syncer-zmtz9","kube-system/konnectivity-agent-gf6wq","openshift-dns/node-resolver-nt2lq"] Sep 08 16:51:10.212023 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.212000 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 16:51:10.212564 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.212547 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.213937 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.213909 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:46:10 +0000 UTC" deadline="2028-02-05 04:22:46.986114034 +0000 UTC" Sep 08 16:51:10.213937 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.213933 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12347h31m36.772183087s" Sep 08 16:51:10.214942 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.214930 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.215383 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215448 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215402 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215448 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215437 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215540 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215456 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215540 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215540 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215672 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj7s\" (UniqueName: \"kubernetes.io/projected/860ebc61-8f2a-4401-829b-f0e1205faf4d-kube-api-access-jgj7s\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.215672 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215638 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 16:51:10.215818 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215800 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7kjj7\"" Sep 08 16:51:10.215878 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215867 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.215930 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.215867 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.218245 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.218207 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.218460 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.218369 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:10.218653 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.218626 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 16:51:10.219121 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.219098 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 16:51:10.219178 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.219113 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5ngt\"" Sep 08 16:51:10.220031 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.220008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:10.220111 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.220077 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:10.222444 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.222426 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.223186 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.223166 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 16:51:10.224785 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.224766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.225099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.225081 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 16:51:10.225367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.225334 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4rwhf\"" Sep 08 16:51:10.225477 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.225438 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.225726 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.225712 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 16:51:10.226879 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.226863 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 16:51:10.226960 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.226878 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 16:51:10.226960 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.226898 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.227190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.227173 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.227413 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.227392 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.227636 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.227619 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.227707 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.227638 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4prjq\"" Sep 08 16:51:10.229856 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.229841 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6c526" Sep 08 16:51:10.230131 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.230101 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.230329 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.230312 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 16:51:10.230390 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.230341 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.230471 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.230456 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4w25c\"" Sep 08 16:51:10.232196 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232176 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.232311 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232218 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.232311 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232221 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pjnt\"" Sep 08 16:51:10.232448 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232370 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.232516 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232500 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 16:51:10.232889 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.232874 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 16:51:10.234877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.234856 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.234974 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.234908 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 16:51:10.235037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.234987 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\"" Sep 08 16:51:10.235488 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.235473 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 16:51:10.237493 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.237467 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.237607 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.237590 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnjkd\"" Sep 08 16:51:10.237707 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.237599 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.237807 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.237695 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.239871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.239849 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:51:10.240013 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.239999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.240064 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.240047 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:10.242943 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.242928 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 16:51:10.243031 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.242972 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 16:51:10.243100 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.243072 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wv25c\"" Sep 08 16:51:10.250130 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.250075 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7t8fl" Sep 08 16:51:10.258974 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.258955 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7t8fl" Sep 08 16:51:10.313267 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.313245 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 16:51:10.315848 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-slash\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.315967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-cnibin\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.315967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315873 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbx4x\" (UniqueName: \"kubernetes.io/projected/830c65e6-ce63-4ecd-87d1-686d43a13d04-kube-api-access-dbx4x\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.315967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-systemd-units\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.315967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.315967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-log-socket\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.315978 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316012 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-netns\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysconfig\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-kubernetes\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316080 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-host\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316101 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/830c65e6-ce63-4ecd-87d1-686d43a13d04-iptables-alerter-script\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.316140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgfq\" (UniqueName: \"kubernetes.io/projected/5acd11f2-9934-413b-ac7a-4646902745f6-kube-api-access-szgfq\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316144 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-socket-dir-parent\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316181 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-etc-kubernetes\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316194 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-run\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316209 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-dbus\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-bin\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-tmp\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316259 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-registration-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316268 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-daemon-config\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-system-cni-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-conf\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316357 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b8082d-9f7d-402c-a13c-04e9d22707ca-host\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316371 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-device-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316381 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cni-binary-copy\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.316449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316393 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-sys-fs\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316404 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316422 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316454 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-kubelet-config\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316472 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316500 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj7s\" (UniqueName: \"kubernetes.io/projected/860ebc61-8f2a-4401-829b-f0e1205faf4d-kube-api-access-jgj7s\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fd9286-b2e0-4f1e-afba-4a87fc36592b-tmp-dir\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316575 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-etc-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316591 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-ovn\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316623 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovn-node-metrics-cert\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316690 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-script-lib\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316765 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316767 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-multus\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.317172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316826 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316852 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-lib-modules\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/2b545246-16c5-445d-844b-6494d032dac3-kube-api-access-6xndr\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-netns\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-systemd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.316985 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317019 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-etc-tuned\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317049 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b8082d-9f7d-402c-a13c-04e9d22707ca-serviceca\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317073 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-systemd\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317099 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317140 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317172 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08fd9286-b2e0-4f1e-afba-4a87fc36592b-hosts-file\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317223 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-k8s-cni-cncf-io\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317265 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-kubelet\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-var-lib-kubelet\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317232 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/860ebc61-8f2a-4401-829b-f0e1205faf4d-socket-dir\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.317871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317340 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-env-overrides\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317369 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71439f0e-1b37-4b69-9473-973fddc608a5-konnectivity-ca\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317394 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-system-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317419 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cnibin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-conf-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317490 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317516 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-modprobe-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317531 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-kubelet\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-var-lib-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317570 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-hostroot\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317585 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-sys\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-config\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r272x\" (UniqueName: \"kubernetes.io/projected/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-kube-api-access-r272x\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317652 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-bin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317680 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-multus-certs\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317697 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4fp\" (UniqueName: \"kubernetes.io/projected/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-kube-api-access-4f4fp\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317729 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.318528 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-netd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317788 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvdp\" (UniqueName: \"kubernetes.io/projected/08fd9286-b2e0-4f1e-afba-4a87fc36592b-kube-api-access-hxvdp\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhwd\" (UniqueName: \"kubernetes.io/projected/cfd00c50-72f9-4af9-964e-bdb82bb89119-kube-api-access-gzhwd\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317840 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-node-log\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317864 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/830c65e6-ce63-4ecd-87d1-686d43a13d04-host-slash\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317911 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71439f0e-1b37-4b69-9473-973fddc608a5-agent-certs\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.319070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.317935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlx57\" (UniqueName: \"kubernetes.io/projected/c6b8082d-9f7d-402c-a13c-04e9d22707ca-kube-api-access-nlx57\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.323460 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.323438 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 16:51:10.326607 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.326591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj7s\" (UniqueName: \"kubernetes.io/projected/860ebc61-8f2a-4401-829b-f0e1205faf4d-kube-api-access-jgj7s\") pod \"aws-ebs-csi-driver-node-s7xgc\" (UID: \"860ebc61-8f2a-4401-829b-f0e1205faf4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.418950 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.418919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-etc-tuned\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.418950 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.418951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b8082d-9f7d-402c-a13c-04e9d22707ca-serviceca\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.418968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-systemd\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.418985 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08fd9286-b2e0-4f1e-afba-4a87fc36592b-hosts-file\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419075 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-systemd\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419087 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08fd9286-b2e0-4f1e-afba-4a87fc36592b-hosts-file\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419090 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-k8s-cni-cncf-io\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-kubelet\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-k8s-cni-cncf-io\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-var-lib-kubelet\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419172 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-kubelet\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419185 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-env-overrides\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419211 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71439f0e-1b37-4b69-9473-973fddc608a5-konnectivity-ca\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-system-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419260 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cnibin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419214 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-var-lib-kubelet\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.419272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419287 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-conf-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419313 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-modprobe-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419384 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-kubelet\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419410 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-var-lib-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-hostroot\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419433 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-system-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419467 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-sys\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419492 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-config\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6b8082d-9f7d-402c-a13c-04e9d22707ca-serviceca\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419520 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r272x\" (UniqueName: \"kubernetes.io/projected/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-kube-api-access-r272x\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-bin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419559 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-conf-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-multus-certs\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4fp\" (UniqueName: \"kubernetes.io/projected/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-kube-api-access-4f4fp\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419643 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-netd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419664 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-sys\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.420147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419385 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cnibin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419668 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvdp\" (UniqueName: \"kubernetes.io/projected/08fd9286-b2e0-4f1e-afba-4a87fc36592b-kube-api-access-hxvdp\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419693 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419521 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-kubelet\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhwd\" (UniqueName: \"kubernetes.io/projected/cfd00c50-72f9-4af9-964e-bdb82bb89119-kube-api-access-gzhwd\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-var-lib-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419775 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71439f0e-1b37-4b69-9473-973fddc608a5-konnectivity-ca\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.419803 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419811 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-node-log\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419818 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-bin\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419849 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-hostroot\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419776 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-node-log\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.419886 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.919858905 +0000 UTC m=+2.064629467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419882 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-multus-certs\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419921 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-netd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419948 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/830c65e6-ce63-4ecd-87d1-686d43a13d04-host-slash\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.421194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.419977 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-modprobe-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420000 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/830c65e6-ce63-4ecd-87d1-686d43a13d04-host-slash\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420040 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71439f0e-1b37-4b69-9473-973fddc608a5-agent-certs\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420068 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlx57\" (UniqueName: \"kubernetes.io/projected/c6b8082d-9f7d-402c-a13c-04e9d22707ca-kube-api-access-nlx57\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420094 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-slash\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420121 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-cnibin\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420148 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbx4x\" (UniqueName: \"kubernetes.io/projected/830c65e6-ce63-4ecd-87d1-686d43a13d04-kube-api-access-dbx4x\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420176 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-systemd-units\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420201 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420206 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-slash\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420225 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-log-socket\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420275 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-netns\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysconfig\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420299 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420338 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-systemd-units\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-log-socket\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-kubernetes\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-config\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-host\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420437 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-cnibin\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420453 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-cni-dir\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420487 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-run-netns\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420488 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-host\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420489 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-kubernetes\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysconfig\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/830c65e6-ce63-4ecd-87d1-686d43a13d04-iptables-alerter-script\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgfq\" (UniqueName: \"kubernetes.io/projected/5acd11f2-9934-413b-ac7a-4646902745f6-kube-api-access-szgfq\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420583 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-socket-dir-parent\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-env-overrides\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-etc-kubernetes\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-run\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420667 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-dbus\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420692 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-bin\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420708 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-socket-dir-parent\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.422877 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420715 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-tmp\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-daemon-config\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420787 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-system-cni-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-conf\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420822 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-etc-kubernetes\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420839 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b8082d-9f7d-402c-a13c-04e9d22707ca-host\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cni-binary-copy\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420923 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420938 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-run\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420949 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-kubelet-config\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420976 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fd9286-b2e0-4f1e-afba-4a87fc36592b-tmp-dir\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-etc-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421056 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-ovn\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421057 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/830c65e6-ce63-4ecd-87d1-686d43a13d04-iptables-alerter-script\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421180 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-conf\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.423578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421195 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6b8082d-9f7d-402c-a13c-04e9d22707ca-host\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421215 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovn-node-metrics-cert\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421245 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-script-lib\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421247 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfd00c50-72f9-4af9-964e-bdb82bb89119-system-cni-dir\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421276 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-multus\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-cni-bin\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421303 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421307 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.420897 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-dbus\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421374 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-etc-openvswitch\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.421458 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.421519 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.921501246 +0000 UTC m=+2.066271808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9853d638-cf47-4f8f-b45f-0a83f02af5d1-kubelet-config\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421676 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08fd9286-b2e0-4f1e-afba-4a87fc36592b-tmp-dir\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-host-var-lib-cni-multus\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.424105 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-ovn\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421867 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-lib-modules\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/2b545246-16c5-445d-844b-6494d032dac3-kube-api-access-6xndr\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421924 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-netns\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421948 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-systemd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.421978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422110 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-etc-sysctl-d\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422162 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-host-run-netns\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422207 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-run-systemd\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b545246-16c5-445d-844b-6494d032dac3-lib-modules\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422714 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422894 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71439f0e-1b37-4b69-9473-973fddc608a5-agent-certs\") pod \"konnectivity-agent-gf6wq\" (UID: \"71439f0e-1b37-4b69-9473-973fddc608a5\") " pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfd00c50-72f9-4af9-964e-bdb82bb89119-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.422935 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-cni-binary-copy\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.423132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-etc-tuned\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.423161 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b545246-16c5-445d-844b-6494d032dac3-tmp\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.423184 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-multus-daemon-config\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.423250 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovnkube-script-lib\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.424760 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.424260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-ovn-node-metrics-cert\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.439000 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.438976 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:10.439000 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.438994 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:10.439000 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.439004 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:10.439251 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.439047 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:10.939034425 +0000 UTC m=+2.083804969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:10.439900 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.439879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvdp\" (UniqueName: \"kubernetes.io/projected/08fd9286-b2e0-4f1e-afba-4a87fc36592b-kube-api-access-hxvdp\") pod \"node-resolver-nt2lq\" (UID: \"08fd9286-b2e0-4f1e-afba-4a87fc36592b\") " pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.441849 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.441756 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4fp\" (UniqueName: \"kubernetes.io/projected/94e7db0e-f57b-41b1-a217-9adbe1b83b4d-kube-api-access-4f4fp\") pod \"multus-6c526\" (UID: \"94e7db0e-f57b-41b1-a217-9adbe1b83b4d\") " pod="openshift-multus/multus-6c526" Sep 08 16:51:10.442096 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.442069 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhwd\" (UniqueName: \"kubernetes.io/projected/cfd00c50-72f9-4af9-964e-bdb82bb89119-kube-api-access-gzhwd\") pod \"multus-additional-cni-plugins-tdpks\" (UID: \"cfd00c50-72f9-4af9-964e-bdb82bb89119\") " pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.444189 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.444160 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r272x\" (UniqueName: \"kubernetes.io/projected/76736d3d-ab1e-4584-a1be-2bb8ec59a2ce-kube-api-access-r272x\") pod \"ovnkube-node-dsbwj\" (UID: \"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.444409 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.444391 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbx4x\" (UniqueName: \"kubernetes.io/projected/830c65e6-ce63-4ecd-87d1-686d43a13d04-kube-api-access-dbx4x\") pod \"iptables-alerter-wh9tv\" (UID: \"830c65e6-ce63-4ecd-87d1-686d43a13d04\") " pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.444732 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.444714 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xndr\" (UniqueName: \"kubernetes.io/projected/2b545246-16c5-445d-844b-6494d032dac3-kube-api-access-6xndr\") pod \"tuned-t6r27\" (UID: \"2b545246-16c5-445d-844b-6494d032dac3\") " pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.444997 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.444980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlx57\" (UniqueName: \"kubernetes.io/projected/c6b8082d-9f7d-402c-a13c-04e9d22707ca-kube-api-access-nlx57\") pod \"node-ca-22bfg\" (UID: \"c6b8082d-9f7d-402c-a13c-04e9d22707ca\") " pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.445095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.445077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgfq\" (UniqueName: \"kubernetes.io/projected/5acd11f2-9934-413b-ac7a-4646902745f6-kube-api-access-szgfq\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.462821 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.462806 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:51:10.536355 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.536270 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" Sep 08 16:51:10.547928 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.547895 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860ebc61_8f2a_4401_829b_f0e1205faf4d.slice/crio-efaac951a16b7f990642ce95175493cdc9cac812438e304a0ed48b113294ffbf WatchSource:0}: Error finding container efaac951a16b7f990642ce95175493cdc9cac812438e304a0ed48b113294ffbf: Status 404 returned error can't find the container with id efaac951a16b7f990642ce95175493cdc9cac812438e304a0ed48b113294ffbf Sep 08 16:51:10.553214 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.553197 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:10.561149 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.561126 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71439f0e_1b37_4b69_9473_973fddc608a5.slice/crio-0d27235987cbd469e61bcc9192f400def9cb953515ba65ac5cb4b41163cb58c6 WatchSource:0}: Error finding container 0d27235987cbd469e61bcc9192f400def9cb953515ba65ac5cb4b41163cb58c6: Status 404 returned error can't find the container with id 0d27235987cbd469e61bcc9192f400def9cb953515ba65ac5cb4b41163cb58c6 Sep 08 16:51:10.562411 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.562396 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:10.570960 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.570942 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76736d3d_ab1e_4584_a1be_2bb8ec59a2ce.slice/crio-721cf05e538eab3ea825daa431f29d7b8117c1dacf3b6a3a54f9501c93687209 WatchSource:0}: Error finding container 721cf05e538eab3ea825daa431f29d7b8117c1dacf3b6a3a54f9501c93687209: Status 404 returned error can't find the container with id 721cf05e538eab3ea825daa431f29d7b8117c1dacf3b6a3a54f9501c93687209 Sep 08 16:51:10.586784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.586764 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nt2lq" Sep 08 16:51:10.595235 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.595215 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fd9286_b2e0_4f1e_afba_4a87fc36592b.slice/crio-04460c7b2f2d1464ff456fb5e26f3dd4756d3e92448275ed39297a5cb51c8bb2 WatchSource:0}: Error finding container 04460c7b2f2d1464ff456fb5e26f3dd4756d3e92448275ed39297a5cb51c8bb2: Status 404 returned error can't find the container with id 04460c7b2f2d1464ff456fb5e26f3dd4756d3e92448275ed39297a5cb51c8bb2 Sep 08 16:51:10.600086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.600067 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:10.600193 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.600178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22bfg" Sep 08 16:51:10.607911 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.607891 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b8082d_9f7d_402c_a13c_04e9d22707ca.slice/crio-2cae4386d83b8bc251e5863a46db34116633fb8bd2f6c6918ba4a68bdd266e3b WatchSource:0}: Error finding container 2cae4386d83b8bc251e5863a46db34116633fb8bd2f6c6918ba4a68bdd266e3b: Status 404 returned error can't find the container with id 2cae4386d83b8bc251e5863a46db34116633fb8bd2f6c6918ba4a68bdd266e3b Sep 08 16:51:10.616914 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.616890 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6c526" Sep 08 16:51:10.624498 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.624475 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7db0e_f57b_41b1_a217_9adbe1b83b4d.slice/crio-d87a6d5cde5cc70a044fc46d46eed82b8cf5f9ad0e7065f6648a4c500304c5dc WatchSource:0}: Error finding container d87a6d5cde5cc70a044fc46d46eed82b8cf5f9ad0e7065f6648a4c500304c5dc: Status 404 returned error can't find the container with id d87a6d5cde5cc70a044fc46d46eed82b8cf5f9ad0e7065f6648a4c500304c5dc Sep 08 16:51:10.638450 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.638429 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdpks" Sep 08 16:51:10.647270 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.647249 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd00c50_72f9_4af9_964e_bdb82bb89119.slice/crio-99d12d455a2e1489af0b82e272335586f6c209a7735683c88256fb283ca6a062 WatchSource:0}: Error finding container 99d12d455a2e1489af0b82e272335586f6c209a7735683c88256fb283ca6a062: Status 404 returned error can't find the container with id 99d12d455a2e1489af0b82e272335586f6c209a7735683c88256fb283ca6a062 Sep 08 16:51:10.650232 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.650216 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6r27" Sep 08 16:51:10.656092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.656074 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wh9tv" Sep 08 16:51:10.657947 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.657923 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b545246_16c5_445d_844b_6494d032dac3.slice/crio-3e704561f7726da8ca3372e953ef6e823a32ffc831f6d6e857f3b3932b8c32e1 WatchSource:0}: Error finding container 3e704561f7726da8ca3372e953ef6e823a32ffc831f6d6e857f3b3932b8c32e1: Status 404 returned error can't find the container with id 3e704561f7726da8ca3372e953ef6e823a32ffc831f6d6e857f3b3932b8c32e1 Sep 08 16:51:10.664213 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:51:10.664179 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830c65e6_ce63_4ecd_87d1_686d43a13d04.slice/crio-c43771824ee29596d4b3dd97169466a7772aa013392f6b17b3824523471b9961 WatchSource:0}: Error finding container c43771824ee29596d4b3dd97169466a7772aa013392f6b17b3824523471b9961: Status 404 returned error can't find the container with id c43771824ee29596d4b3dd97169466a7772aa013392f6b17b3824523471b9961 Sep 08 16:51:10.926930 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.926847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:10.926930 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:10.926891 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:10.927098 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.926989 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.927098 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.926990 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:10.927098 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.927038 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:11.92702395 +0000 UTC m=+3.071794490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:10.927098 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:10.927051 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:11.927045393 +0000 UTC m=+3.071815933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:11.027295 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.027254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:11.027552 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.027466 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:11.027552 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.027496 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:11.027552 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.027512 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:11.027739 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.027572 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:12.027550958 +0000 UTC m=+3.172321515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:11.103719 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.103525 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:11.215667 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.215586 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:11.262300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.262227 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:46:11 +0000 UTC" deadline="2028-02-15 16:45:31.046548093 +0000 UTC" Sep 08 16:51:11.262300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.262298 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12599h54m19.784254731s" Sep 08 16:51:11.337681 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.337642 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:11.337862 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.337803 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:11.344500 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.344447 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wh9tv" event={"ID":"830c65e6-ce63-4ecd-87d1-686d43a13d04","Type":"ContainerStarted","Data":"c43771824ee29596d4b3dd97169466a7772aa013392f6b17b3824523471b9961"} Sep 08 16:51:11.354276 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.351844 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6r27" event={"ID":"2b545246-16c5-445d-844b-6494d032dac3","Type":"ContainerStarted","Data":"3e704561f7726da8ca3372e953ef6e823a32ffc831f6d6e857f3b3932b8c32e1"} Sep 08 16:51:11.354276 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.353776 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerStarted","Data":"99d12d455a2e1489af0b82e272335586f6c209a7735683c88256fb283ca6a062"} Sep 08 16:51:11.355974 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.355285 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22bfg" event={"ID":"c6b8082d-9f7d-402c-a13c-04e9d22707ca","Type":"ContainerStarted","Data":"2cae4386d83b8bc251e5863a46db34116633fb8bd2f6c6918ba4a68bdd266e3b"} Sep 08 16:51:11.356893 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.356868 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nt2lq" event={"ID":"08fd9286-b2e0-4f1e-afba-4a87fc36592b","Type":"ContainerStarted","Data":"04460c7b2f2d1464ff456fb5e26f3dd4756d3e92448275ed39297a5cb51c8bb2"} Sep 08 16:51:11.358828 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.358804 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6c526" event={"ID":"94e7db0e-f57b-41b1-a217-9adbe1b83b4d","Type":"ContainerStarted","Data":"d87a6d5cde5cc70a044fc46d46eed82b8cf5f9ad0e7065f6648a4c500304c5dc"} Sep 08 16:51:11.362185 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.362158 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"721cf05e538eab3ea825daa431f29d7b8117c1dacf3b6a3a54f9501c93687209"} Sep 08 16:51:11.365620 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.365599 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gf6wq" event={"ID":"71439f0e-1b37-4b69-9473-973fddc608a5","Type":"ContainerStarted","Data":"0d27235987cbd469e61bcc9192f400def9cb953515ba65ac5cb4b41163cb58c6"} Sep 08 16:51:11.367161 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.367132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" event={"ID":"860ebc61-8f2a-4401-829b-f0e1205faf4d","Type":"ContainerStarted","Data":"efaac951a16b7f990642ce95175493cdc9cac812438e304a0ed48b113294ffbf"} Sep 08 16:51:11.368967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.368934 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" event={"ID":"56dc1b1e4db5094fc00daf3962293d7e","Type":"ContainerStarted","Data":"0f36ac7d1de02c3da3661e478ccd79a43b3490844d781ad10419989e44d9b55c"} Sep 08 16:51:11.936586 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.936544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:11.936771 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:11.936619 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:11.936771 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.936757 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:11.936908 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.936823 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.93680353 +0000 UTC m=+5.081574088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:11.936908 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.936891 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:11.937037 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:11.936928 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.936916193 +0000 UTC m=+5.081686735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:12.037796 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.037753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:12.037969 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.037944 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:12.037969 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.037964 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:12.038079 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.037979 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:12.038079 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.038044 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:14.038024394 +0000 UTC m=+5.182794956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:12.263215 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.263122 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:46:11 +0000 UTC" deadline="2028-04-01 11:17:20.989169565 +0000 UTC" Sep 08 16:51:12.263215 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.263166 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13698h26m8.726008873s" Sep 08 16:51:12.337573 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.336883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:12.337573 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.336936 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:12.337573 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.337037 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:12.337573 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:12.337530 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:12.388631 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.388542 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" event={"ID":"fed925c216bdbc5803365eb4ce2b6827","Type":"ContainerStarted","Data":"629f77dd0ba65058ae936e7d8906201cdd406ef750f3a0995a75493c4961a117"} Sep 08 16:51:12.977419 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:12.977145 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 16:51:13.339965 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:13.339819 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:13.340436 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:13.339994 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:13.956274 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:13.955534 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:13.956274 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:13.955610 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:13.956274 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:13.955761 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:13.956274 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:13.955822 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:17.955802563 +0000 UTC m=+9.100573122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:13.956274 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:13.956255 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:13.956726 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:13.956310 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:17.956294364 +0000 UTC m=+9.101064920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:14.056830 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:14.056789 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:14.057017 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.056998 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:14.057096 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.057027 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:14.057096 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.057042 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:14.057205 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.057106 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:18.057086266 +0000 UTC m=+9.201856813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:14.337067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:14.336467 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:14.337067 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.336564 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:14.337067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:14.336630 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:14.337067 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:14.336700 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:15.336697 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:15.336268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:15.336697 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:15.336436 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:16.337357 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:16.336661 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:16.337357 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:16.336811 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:16.337357 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:16.337208 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:16.337357 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:16.337306 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:17.337581 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:17.337537 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:17.338059 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:17.337684 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:17.992066 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:17.992027 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:17.992241 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:17.992097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:17.992241 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:17.992230 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:17.992374 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:17.992252 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:17.992374 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:17.992302 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:25.99228514 +0000 UTC m=+17.137055692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:17.992374 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:17.992322 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:25.992310976 +0000 UTC m=+17.137081516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:18.092758 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:18.092693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:18.092950 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.092872 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:18.092950 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.092897 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:18.092950 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.092910 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:18.093203 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.092975 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:26.092956751 +0000 UTC m=+17.237727294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:18.336941 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:18.336849 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:18.337160 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:18.336859 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:18.337160 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.336984 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:18.337160 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:18.337102 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:19.338671 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:19.338183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:19.338671 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:19.338293 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:20.336675 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:20.336639 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:20.336675 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:20.336670 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:20.336936 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:20.336787 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:20.336936 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:20.336886 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:21.336740 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:21.336702 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:21.337342 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:21.336813 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:22.336289 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:22.336255 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:22.336483 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:22.336382 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:22.336483 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:22.336437 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:22.336599 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:22.336538 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:23.336522 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:23.336451 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:23.336922 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:23.336588 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:24.337068 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:24.337039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:24.337525 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:24.337153 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:24.337525 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:24.337232 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:24.337525 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:24.337339 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:25.337023 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:25.336982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:25.337206 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:25.337105 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:26.053556 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:26.053517 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:26.053735 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:26.053608 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:26.053735 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.053690 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:26.053849 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.053754 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:26.053849 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.053769 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.053749419 +0000 UTC m=+33.198519977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:26.053849 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.053806 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.053791089 +0000 UTC m=+33.198561655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:26.154081 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:26.154042 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:26.154254 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.154233 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:26.154312 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.154261 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:26.154312 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.154275 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:26.154440 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.154354 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:42.154334979 +0000 UTC m=+33.299105534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:26.337087 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:26.337013 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:26.337224 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:26.337013 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:26.337224 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.337138 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:26.337561 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:26.337221 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:27.336300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:27.336265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:27.336486 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:27.336413 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:28.336261 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:28.336223 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:28.336782 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:28.336223 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:28.336782 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:28.336345 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:28.336782 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:28.336487 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:29.337522 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:29.337483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:29.337945 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:29.337606 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:30.336544 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.336514 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:30.336661 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:30.336638 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:30.336735 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.336681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:30.336838 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:30.336808 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:30.455536 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.455250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" event={"ID":"56dc1b1e4db5094fc00daf3962293d7e","Type":"ContainerStarted","Data":"88b305684b79861149b24cfc0b30c60297b502ba0c116226bf38293484b96396"} Sep 08 16:51:30.457432 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.457407 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6r27" event={"ID":"2b545246-16c5-445d-844b-6494d032dac3","Type":"ContainerStarted","Data":"e477e53df51f9a2ea31c80b5affe3ef8f377f1d85a97c97d697aa18b78c4373b"} Sep 08 16:51:30.460292 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.460264 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6c526" event={"ID":"94e7db0e-f57b-41b1-a217-9adbe1b83b4d","Type":"ContainerStarted","Data":"3b0ebdd98d6f4b94653c4db1f52999c797d54ef02a273b2061da5dac2c30683a"} Sep 08 16:51:30.462448 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.462383 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"da623780b4d321e21d877a0d34e50b66e4bf314fa9c8e5eee45af461ff6c0c53"} Sep 08 16:51:30.469763 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.469715 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-198.ec2.internal" podStartSLOduration=19.469698486 podStartE2EDuration="19.469698486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:30.469135945 +0000 UTC m=+21.613906507" watchObservedRunningTime="2026-09-08 16:51:30.469698486 +0000 UTC m=+21.614469056" Sep 08 16:51:30.485128 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.484888 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6c526" podStartSLOduration=0.80617251 podStartE2EDuration="20.484869741s" totalImagesPullingTime="19.678697231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:30.484670766 +0000 UTC m=+21.629441328" watchObservedRunningTime="2026-09-08 16:51:30.484869741 +0000 UTC m=+21.629640303" Sep 08 16:51:30.503232 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:30.503149 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t6r27" podStartSLOduration=0.914642146 podStartE2EDuration="20.503136771s" totalImagesPullingTime="19.588494625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:30.503000798 +0000 UTC m=+21.647771361" watchObservedRunningTime="2026-09-08 16:51:30.503136771 +0000 UTC m=+21.647907331" Sep 08 16:51:31.337339 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.337090 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:31.337510 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:31.337460 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:31.465863 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.465828 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gf6wq" event={"ID":"71439f0e-1b37-4b69-9473-973fddc608a5","Type":"ContainerStarted","Data":"417b1419001d0176514878f760c9d9ba63a14458ba2fc642d227a608297bc477"} Sep 08 16:51:31.467245 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.467223 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" event={"ID":"860ebc61-8f2a-4401-829b-f0e1205faf4d","Type":"ContainerStarted","Data":"ecfb1e82a7110a5cc5dad2bba8aa4491f81e8c9102d6036fbe67b9779bdcd79c"} Sep 08 16:51:31.468635 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.468606 2556 generic.go:356] "Generic (PLEG): container finished" podID="fed925c216bdbc5803365eb4ce2b6827" containerID="1506d65c829778ffcd2504c55a14c0b9c762fc1c36b423adca8f82cfc672107c" exitCode=0 Sep 08 16:51:31.468723 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.468689 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" event={"ID":"fed925c216bdbc5803365eb4ce2b6827","Type":"ContainerDied","Data":"1506d65c829778ffcd2504c55a14c0b9c762fc1c36b423adca8f82cfc672107c"} Sep 08 16:51:31.470219 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.470199 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="d5df48ae062cbf43260bb284a51eb8a2e93ad55726c7e423dbc1f782485eb558" exitCode=0 Sep 08 16:51:31.470296 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.470259 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"d5df48ae062cbf43260bb284a51eb8a2e93ad55726c7e423dbc1f782485eb558"} Sep 08 16:51:31.471714 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.471697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22bfg" event={"ID":"c6b8082d-9f7d-402c-a13c-04e9d22707ca","Type":"ContainerStarted","Data":"7412e435d32f7f2b0365dc8656c0f3641485d89058dbe803e95e88dcf03b88e7"} Sep 08 16:51:31.473348 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.473312 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nt2lq" event={"ID":"08fd9286-b2e0-4f1e-afba-4a87fc36592b","Type":"ContainerStarted","Data":"5a4dda48eaf630fe5e171a1d21b2d818459ff3b84a3131b69072ae6b94d849e3"} Sep 08 16:51:31.476162 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.476137 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"c1ce1d1b960b0a764b41aa5a4315480e644601df1a29de0dfaf79a3e3c5e95bd"} Sep 08 16:51:31.476248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.476218 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"36eba9309c53badbfdf15a9641b2aef9d031b3c5d7f8c1a97c831e06112765a9"} Sep 08 16:51:31.476287 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.476273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"f3aeaa544f05b2c09600cd5929a982284426049cddf1371fa0e513e89ad00127"} Sep 08 16:51:31.476356 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.476344 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"39c437736890dffc1c0ca8c9572e3af1b596cd8879e8ce763c588a80af879cc7"} Sep 08 16:51:31.476405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.476393 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"e88297e8d59f8f5c9e98f2be622c4034f9945f734ad5654bd7f439ca2e97ff09"} Sep 08 16:51:31.480681 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.480646 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gf6wq" podStartSLOduration=1.810908561 podStartE2EDuration="21.480635447s" totalImagesPullingTime="19.669726886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:31.480136313 +0000 UTC m=+22.624906873" watchObservedRunningTime="2026-09-08 16:51:31.480635447 +0000 UTC m=+22.625406008" Sep 08 16:51:31.530834 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.530761 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nt2lq" podStartSLOduration=1.8981854390000001 podStartE2EDuration="21.530746698s" totalImagesPullingTime="19.632561259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:31.5300295 +0000 UTC m=+22.674800063" watchObservedRunningTime="2026-09-08 16:51:31.530746698 +0000 UTC m=+22.675517258" Sep 08 16:51:31.543594 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:31.543559 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-22bfg" podStartSLOduration=1.92469899 podStartE2EDuration="21.543533523s" totalImagesPullingTime="19.618834533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:31.543235158 +0000 UTC m=+22.688005720" watchObservedRunningTime="2026-09-08 16:51:31.543533523 +0000 UTC m=+22.688304084" Sep 08 16:51:32.204050 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.204014 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 16:51:32.282552 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.282462 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:51:32.204032069Z","UUID":"74e561ed-440c-49a9-8c79-0c9e1e921a30","Handler":null,"Name":"","Endpoint":""} Sep 08 16:51:32.283790 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.283774 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 16:51:32.283878 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.283801 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 16:51:32.336202 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.336137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:32.336202 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.336168 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:32.336452 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:32.336257 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:32.336452 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:32.336351 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:32.479998 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.479960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" event={"ID":"860ebc61-8f2a-4401-829b-f0e1205faf4d","Type":"ContainerStarted","Data":"7cfe325e606463ab525db793f74d42f704b93ce7d7533188f7d2b5ce2f01d19e"} Sep 08 16:51:32.482136 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.482112 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" event={"ID":"fed925c216bdbc5803365eb4ce2b6827","Type":"ContainerStarted","Data":"c4f40d2b791f844f5019afed6a29e93d46822b07b410c1f513bd06a467b4c2f4"} Sep 08 16:51:32.483797 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.483750 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wh9tv" event={"ID":"830c65e6-ce63-4ecd-87d1-686d43a13d04","Type":"ContainerStarted","Data":"f8f3544d71b8166a4303f5f8b2d0c053b7b61ce64f813e1e3b27a920b51177f2"} Sep 08 16:51:32.497683 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.497640 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-198.ec2.internal" podStartSLOduration=21.497628323 podStartE2EDuration="21.497628323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.49697582 +0000 UTC m=+23.641746381" watchObservedRunningTime="2026-09-08 16:51:32.497628323 +0000 UTC m=+23.642398883" Sep 08 16:51:32.511889 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:32.511854 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wh9tv" podStartSLOduration=2.933750028 podStartE2EDuration="22.511842908s" totalImagesPullingTime="19.57809288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:32.511614394 +0000 UTC m=+23.656384956" watchObservedRunningTime="2026-09-08 16:51:32.511842908 +0000 UTC m=+23.656613471" Sep 08 16:51:33.338644 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:33.336588 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:33.338644 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:33.336721 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:33.489194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:33.489165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"3514afb0e8aca29838a6b5fc6c796e1946c953cce07bf342b28288dd8c32889a"} Sep 08 16:51:34.053425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.053396 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:34.053993 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.053975 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:34.336340 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.336295 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:34.336340 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.336311 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:34.336560 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:34.336427 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:34.336608 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:34.336559 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:34.494571 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.494535 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" event={"ID":"860ebc61-8f2a-4401-829b-f0e1205faf4d","Type":"ContainerStarted","Data":"a296300a44359fb4fb6b0e312df1de4bbf58cff8d0ff4260074ac9d69b265f4d"} Sep 08 16:51:34.494981 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.494712 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:34.495273 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.495255 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gf6wq" Sep 08 16:51:34.517149 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:34.517102 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s7xgc" podStartSLOduration=1.8805992740000002 podStartE2EDuration="24.517090068s" totalImagesPullingTime="22.636490794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:34.516813577 +0000 UTC m=+25.661584139" watchObservedRunningTime="2026-09-08 16:51:34.517090068 +0000 UTC m=+25.661860628" Sep 08 16:51:35.246774 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:35.246748 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nt2lq_08fd9286-b2e0-4f1e-afba-4a87fc36592b/dns-node-resolver/0.log" Sep 08 16:51:35.336895 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:35.336861 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:35.337050 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:35.336993 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:36.025953 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.025927 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22bfg_c6b8082d-9f7d-402c-a13c-04e9d22707ca/node-ca/0.log" Sep 08 16:51:36.336419 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.336351 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:36.336419 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.336355 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:36.336575 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:36.336438 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:36.336575 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:36.336503 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:36.503440 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.503347 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" event={"ID":"76736d3d-ab1e-4584-a1be-2bb8ec59a2ce","Type":"ContainerStarted","Data":"9ce2c635b64e5c30bebd55da493ea8254578ac97cc5691942e0cd3e505cd5e0e"} Sep 08 16:51:36.503722 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.503662 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:36.503771 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.503740 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:36.521162 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.521075 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:36.522179 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.521889 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:36.542265 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:36.542221 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" podStartSLOduration=6.823549529 podStartE2EDuration="26.542207412s" totalImagesPullingTime="19.718657883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:36.541988069 +0000 UTC m=+27.686758629" watchObservedRunningTime="2026-09-08 16:51:36.542207412 +0000 UTC m=+27.686977994" Sep 08 16:51:37.337032 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:37.336798 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:37.337480 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:37.337068 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:37.506975 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:37.506945 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="3286a1e42638c258934d38e74396af804b3cceeb9c3c8a5baf00f9c4773d4d11" exitCode=0 Sep 08 16:51:37.507111 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:37.507031 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"3286a1e42638c258934d38e74396af804b3cceeb9c3c8a5baf00f9c4773d4d11"} Sep 08 16:51:37.507111 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:37.507080 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 16:51:37.931027 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:37.931002 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:51:38.336804 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:38.336733 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:38.336804 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:38.336766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:38.337054 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:38.336848 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:38.337054 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:38.336962 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:39.337412 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:39.337381 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:39.337886 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:39.337501 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:39.513495 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:39.513463 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="a93a6ed9904cb743ae1aee304b7c690537b38eff7ad10d43b5cc4e62de3dc373" exitCode=0 Sep 08 16:51:39.513642 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:39.513550 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"a93a6ed9904cb743ae1aee304b7c690537b38eff7ad10d43b5cc4e62de3dc373"} Sep 08 16:51:40.337067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:40.337034 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:40.337253 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:40.337119 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:40.337253 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:40.337175 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:40.337391 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:40.337294 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:41.336452 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:41.336416 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:41.336927 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:41.336545 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:41.521103 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:41.521069 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="c16ed012398d36bff785cfbb8b18101ffb4fd2be001ab4f17d6c90e98922a785" exitCode=0 Sep 08 16:51:41.521297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:41.521130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"c16ed012398d36bff785cfbb8b18101ffb4fd2be001ab4f17d6c90e98922a785"} Sep 08 16:51:42.075698 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:42.075663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:42.075698 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:42.075713 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:42.075905 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.075816 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:42.075905 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.075863 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs podName:5acd11f2-9934-413b-ac7a-4646902745f6 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:14.075848764 +0000 UTC m=+65.220619303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs") pod "network-metrics-daemon-j74bn" (UID: "5acd11f2-9934-413b-ac7a-4646902745f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 16:51:42.075983 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.075902 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:42.075983 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.075951 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret podName:9853d638-cf47-4f8f-b45f-0a83f02af5d1 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:14.075939854 +0000 UTC m=+65.220710393 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret") pod "global-pull-secret-syncer-zmtz9" (UID: "9853d638-cf47-4f8f-b45f-0a83f02af5d1") : object "kube-system"/"original-pull-secret" not registered Sep 08 16:51:42.176850 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:42.176813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:42.177021 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.176945 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 16:51:42.177021 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.176958 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 16:51:42.177021 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.176967 2556 projected.go:196] Error preparing data for projected volume kube-api-access-wjjr7 for pod openshift-network-diagnostics/network-check-target-sdrwc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:42.177021 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.177010 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7 podName:7dd46390-5e12-4101-9a78-8a3444ed9d87 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:14.176998543 +0000 UTC m=+65.321769082 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wjjr7" (UniqueName: "kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7") pod "network-check-target-sdrwc" (UID: "7dd46390-5e12-4101-9a78-8a3444ed9d87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 16:51:42.336379 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:42.336303 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:42.336526 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.336408 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:42.336526 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:42.336459 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:42.336939 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:42.336539 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:43.336546 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:43.336513 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:43.336997 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:43.336617 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:44.336961 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:44.336923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:44.336961 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:44.336939 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:44.337574 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:44.337043 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:44.337574 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:44.337162 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:45.340456 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:45.340427 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:45.340884 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:45.340534 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:46.336881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:46.336846 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:46.336881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:46.336888 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:46.337100 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:46.336975 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:46.337147 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:46.337102 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:47.339415 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:47.339347 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:47.339844 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:47.339444 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:47.536911 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:47.536879 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="bc604845687b4777d521a711ad34dd0aea91ba0bfc628e7ef8197604bcaf7194" exitCode=0 Sep 08 16:51:47.537058 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:47.536929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"bc604845687b4777d521a711ad34dd0aea91ba0bfc628e7ef8197604bcaf7194"} Sep 08 16:51:48.336234 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:48.336199 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:48.336410 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:48.336304 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:48.336410 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:48.336357 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:48.336484 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:48.336451 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:48.541886 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:48.541854 2556 generic.go:356] "Generic (PLEG): container finished" podID="cfd00c50-72f9-4af9-964e-bdb82bb89119" containerID="4bb4c91b2f073018b3f425663e3d89e73d6da249b43b990931cf43c8663430c9" exitCode=0 Sep 08 16:51:48.542239 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:48.541907 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerDied","Data":"4bb4c91b2f073018b3f425663e3d89e73d6da249b43b990931cf43c8663430c9"} Sep 08 16:51:49.337578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:49.337408 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:49.337736 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:49.337637 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:49.547449 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:49.547414 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdpks" event={"ID":"cfd00c50-72f9-4af9-964e-bdb82bb89119","Type":"ContainerStarted","Data":"ed22ed34999d038e63363bdc561f9b67f312791d06f41e8eba80117a67726c8b"} Sep 08 16:51:49.569617 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:49.569579 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tdpks" podStartSLOduration=10.888888404 podStartE2EDuration="39.56954835s" totalImagesPullingTime="28.680659946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:49.569370283 +0000 UTC m=+40.714140844" watchObservedRunningTime="2026-09-08 16:51:49.56954835 +0000 UTC m=+40.714318910" Sep 08 16:51:50.336295 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:50.336268 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:50.336473 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:50.336372 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:50.336473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:50.336430 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:50.336549 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:50.336533 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:51.336995 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:51.336956 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:51.337405 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:51.337089 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:52.336953 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:52.336924 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:52.337115 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:52.337014 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:52.337115 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:52.337039 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:52.337115 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:52.337092 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:53.336604 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:53.336571 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:53.336783 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:53.336705 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:54.336417 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:54.336387 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:54.336803 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:54.336394 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:54.336803 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:54.336494 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:54.336803 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:54.336547 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:55.337012 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:55.336980 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:55.337389 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:55.337082 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:56.336071 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:56.336042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:56.336280 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:56.336141 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:56.336280 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:56.336214 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:56.336373 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:56.336311 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:57.336291 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:57.336248 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:57.336683 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:57.336394 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:58.336665 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:58.336631 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:58.337117 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:58.336757 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:58.337117 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:58.336806 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:58.337117 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:58.336896 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:59.339332 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.339297 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:59.339704 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:59.339456 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:51:59.370761 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.370735 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmtz9"] Sep 08 16:51:59.374912 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.374885 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdrwc"] Sep 08 16:51:59.375136 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.375080 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j74bn"] Sep 08 16:51:59.375805 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.375148 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:51:59.375805 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:59.375243 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:51:59.375805 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.375305 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:51:59.375805 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:59.375385 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:51:59.568473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:51:59.568399 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:51:59.568608 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:51:59.568493 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:52:01.336778 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:01.336750 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:01.336778 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:01.336782 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:01.337367 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:01.336846 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:52:01.337367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:01.336754 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:01.337367 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:01.336958 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:52:01.337367 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:01.337067 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:52:03.336360 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:03.336210 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:03.336768 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:03.336208 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:03.336768 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:03.336430 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sdrwc" podUID="7dd46390-5e12-4101-9a78-8a3444ed9d87" Sep 08 16:52:03.336768 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:03.336219 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:03.336768 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:03.336516 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j74bn" podUID="5acd11f2-9934-413b-ac7a-4646902745f6" Sep 08 16:52:03.336768 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:03.336568 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmtz9" podUID="9853d638-cf47-4f8f-b45f-0a83f02af5d1" Sep 08 16:52:04.138738 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.138715 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-198.ec2.internal" event="NodeReady" Sep 08 16:52:04.138878 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.138828 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 16:52:04.187004 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.186973 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xgx5g"] Sep 08 16:52:04.188711 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.188695 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.191445 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.191421 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 16:52:04.191445 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.191439 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 16:52:04.191623 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.191507 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qdttq\"" Sep 08 16:52:04.191986 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.191968 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 16:52:04.196926 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.196904 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hw7h4"] Sep 08 16:52:04.198748 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.198567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.200147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.200130 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xgx5g"] Sep 08 16:52:04.201586 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.201557 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 16:52:04.201690 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.201672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxh7j\"" Sep 08 16:52:04.201895 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.201879 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 16:52:04.201971 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.201901 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 16:52:04.201971 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.201957 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 16:52:04.211524 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.211503 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hw7h4"] Sep 08 16:52:04.245177 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245154 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.245317 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnptm\" (UniqueName: \"kubernetes.io/projected/bd0702a9-c059-4dcf-91ec-e204eada2bd9-kube-api-access-hnptm\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.245317 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245208 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xqr\" (UniqueName: \"kubernetes.io/projected/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-api-access-29xqr\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.245317 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245227 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7038167-6504-4a28-ba03-49fdb45ab0b4-crio-socket\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.245317 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7038167-6504-4a28-ba03-49fdb45ab0b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.245530 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245366 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd0702a9-c059-4dcf-91ec-e204eada2bd9-cert\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.245530 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.245420 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7038167-6504-4a28-ba03-49fdb45ab0b4-data-volume\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.289313 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.289289 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xc9ml"] Sep 08 16:52:04.291085 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.291070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.293933 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.293916 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpm9l\"" Sep 08 16:52:04.294191 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.294172 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 16:52:04.294724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.294705 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 16:52:04.302978 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.302957 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xc9ml"] Sep 08 16:52:04.346466 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7038167-6504-4a28-ba03-49fdb45ab0b4-data-volume\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-config-volume\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-metrics-tls\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346627 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4nh\" (UniqueName: \"kubernetes.io/projected/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-kube-api-access-nk4nh\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346672 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnptm\" (UniqueName: \"kubernetes.io/projected/bd0702a9-c059-4dcf-91ec-e204eada2bd9-kube-api-access-hnptm\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xqr\" (UniqueName: \"kubernetes.io/projected/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-api-access-29xqr\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346736 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346722 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-tmp-dir\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.346987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346744 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7038167-6504-4a28-ba03-49fdb45ab0b4-crio-socket\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7038167-6504-4a28-ba03-49fdb45ab0b4-data-volume\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7038167-6504-4a28-ba03-49fdb45ab0b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.346987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd0702a9-c059-4dcf-91ec-e204eada2bd9-cert\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.346987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.346923 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7038167-6504-4a28-ba03-49fdb45ab0b4-crio-socket\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.347157 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.347054 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.350500 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.350483 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd0702a9-c059-4dcf-91ec-e204eada2bd9-cert\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.350571 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.350501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7038167-6504-4a28-ba03-49fdb45ab0b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.354764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.354739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xqr\" (UniqueName: \"kubernetes.io/projected/b7038167-6504-4a28-ba03-49fdb45ab0b4-kube-api-access-29xqr\") pod \"insights-runtime-extractor-hw7h4\" (UID: \"b7038167-6504-4a28-ba03-49fdb45ab0b4\") " pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.355502 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.355479 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnptm\" (UniqueName: \"kubernetes.io/projected/bd0702a9-c059-4dcf-91ec-e204eada2bd9-kube-api-access-hnptm\") pod \"ingress-canary-xgx5g\" (UID: \"bd0702a9-c059-4dcf-91ec-e204eada2bd9\") " pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.448163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448073 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-config-volume\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.448163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-metrics-tls\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.448163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448124 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4nh\" (UniqueName: \"kubernetes.io/projected/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-kube-api-access-nk4nh\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.448163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-tmp-dir\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.448508 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448494 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-tmp-dir\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.448641 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.448615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-config-volume\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.450398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.450380 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-metrics-tls\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.456385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.456365 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4nh\" (UniqueName: \"kubernetes.io/projected/2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3-kube-api-access-nk4nh\") pod \"dns-default-xc9ml\" (UID: \"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3\") " pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.499037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.499012 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xgx5g" Sep 08 16:52:04.507680 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.507657 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hw7h4" Sep 08 16:52:04.599235 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.599211 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:04.776518 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.776443 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xgx5g"] Sep 08 16:52:04.779396 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.779374 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hw7h4"] Sep 08 16:52:04.782745 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:04.782725 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7038167_6504_4a28_ba03_49fdb45ab0b4.slice/crio-31a09782d10dfecef847cd9f4b731aa7f58ea5616d0c06d975e286f538495c69 WatchSource:0}: Error finding container 31a09782d10dfecef847cd9f4b731aa7f58ea5616d0c06d975e286f538495c69: Status 404 returned error can't find the container with id 31a09782d10dfecef847cd9f4b731aa7f58ea5616d0c06d975e286f538495c69 Sep 08 16:52:04.853661 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.853639 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-q5fvg"] Sep 08 16:52:04.855517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.855501 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:04.859180 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.859160 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 08 16:52:04.859699 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.859684 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-stzrh\"" Sep 08 16:52:04.860769 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.860747 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 16:52:04.860934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.860789 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 08 16:52:04.860934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.860811 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 16:52:04.860934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.860832 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 16:52:04.871271 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.871250 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-q5fvg"] Sep 08 16:52:04.878481 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.878455 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xc9ml"] Sep 08 16:52:04.883060 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:04.883025 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acadec1_8dde_4dea_bb5a_5eb7b2f6a6e3.slice/crio-157e202736739123efa30e2eeeabcd755338714ce0c0d8758e811005237cb3d6 WatchSource:0}: Error finding container 157e202736739123efa30e2eeeabcd755338714ce0c0d8758e811005237cb3d6: Status 404 returned error can't find the container with id 157e202736739123efa30e2eeeabcd755338714ce0c0d8758e811005237cb3d6 Sep 08 16:52:04.952225 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.952190 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltxb\" (UniqueName: \"kubernetes.io/projected/2540541d-0d48-46f1-acd5-2bf9a2506373-kube-api-access-zltxb\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:04.952408 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.952259 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:04.952408 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.952299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:04.952408 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:04.952362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2540541d-0d48-46f1-acd5-2bf9a2506373-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.053080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.052989 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.053080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.053025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.053080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.053049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2540541d-0d48-46f1-acd5-2bf9a2506373-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.053377 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.053098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltxb\" (UniqueName: \"kubernetes.io/projected/2540541d-0d48-46f1-acd5-2bf9a2506373-kube-api-access-zltxb\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.053377 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:05.053141 2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 08 16:52:05.053377 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:05.053220 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls podName:2540541d-0d48-46f1-acd5-2bf9a2506373 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:05.553203257 +0000 UTC m=+56.697973819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-q5fvg" (UID: "2540541d-0d48-46f1-acd5-2bf9a2506373") : secret "prometheus-operator-tls" not found Sep 08 16:52:05.053832 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.053815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2540541d-0d48-46f1-acd5-2bf9a2506373-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.055366 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.055343 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.061348 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.061312 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltxb\" (UniqueName: \"kubernetes.io/projected/2540541d-0d48-46f1-acd5-2bf9a2506373-kube-api-access-zltxb\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.340872 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.340794 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:05.340872 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.340824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:05.341075 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.340795 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:05.343912 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.343885 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:52:05.344086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.343919 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:52:05.344086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.343990 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\"" Sep 08 16:52:05.344086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.344076 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8582r\"" Sep 08 16:52:05.344293 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.344195 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:52:05.344386 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.344286 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:52:05.557215 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.557183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.560147 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.560122 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2540541d-0d48-46f1-acd5-2bf9a2506373-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-q5fvg\" (UID: \"2540541d-0d48-46f1-acd5-2bf9a2506373\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:05.583836 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.583799 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xc9ml" event={"ID":"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3","Type":"ContainerStarted","Data":"157e202736739123efa30e2eeeabcd755338714ce0c0d8758e811005237cb3d6"} Sep 08 16:52:05.585380 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.585301 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xgx5g" event={"ID":"bd0702a9-c059-4dcf-91ec-e204eada2bd9","Type":"ContainerStarted","Data":"00c3ff65134e9bd644574b5829e6ff8deb20db38b26ba39aa7b1335169f8373b"} Sep 08 16:52:05.587401 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.587375 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hw7h4" event={"ID":"b7038167-6504-4a28-ba03-49fdb45ab0b4","Type":"ContainerStarted","Data":"781d7568859d70e5390d78440ce6bfa1e8fcb5558430ff721ab93c519819257b"} Sep 08 16:52:05.587489 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.587435 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hw7h4" event={"ID":"b7038167-6504-4a28-ba03-49fdb45ab0b4","Type":"ContainerStarted","Data":"31a09782d10dfecef847cd9f4b731aa7f58ea5616d0c06d975e286f538495c69"} Sep 08 16:52:05.764577 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:05.764542 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" Sep 08 16:52:06.037583 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:06.037510 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-q5fvg"] Sep 08 16:52:06.042014 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:06.041982 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2540541d_0d48_46f1_acd5_2bf9a2506373.slice/crio-b88b076b324d28cb034800f36faa90b698c07ac2f9a005742ff95da7125cc8d4 WatchSource:0}: Error finding container b88b076b324d28cb034800f36faa90b698c07ac2f9a005742ff95da7125cc8d4: Status 404 returned error can't find the container with id b88b076b324d28cb034800f36faa90b698c07ac2f9a005742ff95da7125cc8d4 Sep 08 16:52:06.591839 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:06.591798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" event={"ID":"2540541d-0d48-46f1-acd5-2bf9a2506373","Type":"ContainerStarted","Data":"b88b076b324d28cb034800f36faa90b698c07ac2f9a005742ff95da7125cc8d4"} Sep 08 16:52:06.594374 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:06.594345 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hw7h4" event={"ID":"b7038167-6504-4a28-ba03-49fdb45ab0b4","Type":"ContainerStarted","Data":"be31b7df47558274be4cff0de3422d3cb442ebca754e3ef281d4c21725c4d778"} Sep 08 16:52:09.523910 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.523834 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsbwj" Sep 08 16:52:09.604661 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.604625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xc9ml" event={"ID":"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3","Type":"ContainerStarted","Data":"b3aa6a99e47e3315a560baa6f6f68e2a7bbd067260251ce56f3c74423f4fdb2a"} Sep 08 16:52:09.604863 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.604787 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:09.604863 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.604816 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xc9ml" event={"ID":"2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3","Type":"ContainerStarted","Data":"c665d34b03fb979326b50dd3f1d7a70d7c503b5fc0c66a92132950de31ac0138"} Sep 08 16:52:09.605967 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.605942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xgx5g" event={"ID":"bd0702a9-c059-4dcf-91ec-e204eada2bd9","Type":"ContainerStarted","Data":"138464cb3e001c4da3d326c1082e751decc368325bdaf5314a7b95176937e346"} Sep 08 16:52:09.607599 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.607581 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" event={"ID":"2540541d-0d48-46f1-acd5-2bf9a2506373","Type":"ContainerStarted","Data":"6b6ba2e41574ecb784bce669d06a654f00decec4673495a8dc18227252747dce"} Sep 08 16:52:09.607683 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.607614 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" event={"ID":"2540541d-0d48-46f1-acd5-2bf9a2506373","Type":"ContainerStarted","Data":"42902c1d3cc10e1f639b95da849194306ee645001e860f98622643c33f0d12c8"} Sep 08 16:52:09.609597 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.609578 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hw7h4" event={"ID":"b7038167-6504-4a28-ba03-49fdb45ab0b4","Type":"ContainerStarted","Data":"e7189e59322b7095f72b567fc911cd821fa96dfdfd52d71eab223dd09c236a01"} Sep 08 16:52:09.623567 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.623515 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xc9ml" podStartSLOduration=1.447595288 podStartE2EDuration="5.6235015s" totalImagesPullingTime="4.175906212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:09.622599342 +0000 UTC m=+60.767369903" watchObservedRunningTime="2026-09-08 16:52:09.6235015 +0000 UTC m=+60.768272061" Sep 08 16:52:09.643074 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.643031 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-q5fvg" podStartSLOduration=2.600297811 podStartE2EDuration="5.643021279s" totalImagesPullingTime="3.042723468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:09.642017785 +0000 UTC m=+60.786788358" watchObservedRunningTime="2026-09-08 16:52:09.643021279 +0000 UTC m=+60.787791839" Sep 08 16:52:09.657305 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.657254 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xgx5g" podStartSLOduration=1.712940155 podStartE2EDuration="5.657239027s" totalImagesPullingTime="3.944298872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:09.657084068 +0000 UTC m=+60.801854630" watchObservedRunningTime="2026-09-08 16:52:09.657239027 +0000 UTC m=+60.802009589" Sep 08 16:52:09.679046 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:09.679000 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hw7h4" podStartSLOduration=1.876564078 podStartE2EDuration="5.678985998s" totalImagesPullingTime="3.80242192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:09.677938354 +0000 UTC m=+60.822708931" watchObservedRunningTime="2026-09-08 16:52:09.678985998 +0000 UTC m=+60.823756559" Sep 08 16:52:11.192990 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.192954 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk"] Sep 08 16:52:11.195044 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.195024 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.197589 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.197569 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rfpr6\"" Sep 08 16:52:11.197780 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.197762 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 08 16:52:11.197942 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.197922 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 08 16:52:11.210026 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.210007 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk"] Sep 08 16:52:11.214144 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.214120 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2zjlf"] Sep 08 16:52:11.216257 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.216238 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs"] Sep 08 16:52:11.216421 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.216402 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.218303 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.218286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.219065 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.219048 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 16:52:11.219245 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.219225 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 16:52:11.219359 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.219270 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 16:52:11.219359 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.219273 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-crsks\"" Sep 08 16:52:11.220759 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.220743 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 08 16:52:11.220759 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.220761 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sfgmc\"" Sep 08 16:52:11.221054 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.221038 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 08 16:52:11.221109 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.221100 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 08 16:52:11.231901 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.231877 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs"] Sep 08 16:52:11.300145 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300113 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-wtmp\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300145 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300144 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7514196b-f24b-4e94-a455-379adc691f79-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300203 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300287 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-metrics-client-ca\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300305 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qql6\" (UniqueName: \"kubernetes.io/projected/6c466071-335d-4f93-92d8-2830b8f01b18-kube-api-access-8qql6\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300337 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300385 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-tls\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300431 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-textfile\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300470 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-root\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300498 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-accelerators-collector-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300540 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300566 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99m7\" (UniqueName: \"kubernetes.io/projected/02d58084-9a2b-49c6-a79e-0d09c632d179-kube-api-access-n99m7\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300595 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lljw\" (UniqueName: \"kubernetes.io/projected/7514196b-f24b-4e94-a455-379adc691f79-kube-api-access-2lljw\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300623 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02d58084-9a2b-49c6-a79e-0d09c632d179-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300680 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.300784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.300758 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-sys\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401071 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-textfile\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401071 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-root\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401092 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-accelerators-collector-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401112 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401134 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99m7\" (UniqueName: \"kubernetes.io/projected/02d58084-9a2b-49c6-a79e-0d09c632d179-kube-api-access-n99m7\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401158 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lljw\" (UniqueName: \"kubernetes.io/projected/7514196b-f24b-4e94-a455-379adc691f79-kube-api-access-2lljw\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401171 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-root\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401223 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02d58084-9a2b-49c6-a79e-0d09c632d179-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.401298 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401358 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-sys\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401419 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-sys\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-wtmp\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401499 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401511 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-textfile\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401527 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7514196b-f24b-4e94-a455-379adc691f79-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401607 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401638 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401643 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-wtmp\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.401688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401682 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-metrics-client-ca\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qql6\" (UniqueName: \"kubernetes.io/projected/6c466071-335d-4f93-92d8-2830b8f01b18-kube-api-access-8qql6\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401763 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-tls\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-accelerators-collector-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.401933 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7514196b-f24b-4e94-a455-379adc691f79-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.402531 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.402356 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.402590 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.402554 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c466071-335d-4f93-92d8-2830b8f01b18-metrics-client-ca\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.402892 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.402827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02d58084-9a2b-49c6-a79e-0d09c632d179-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.403043 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.403020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.404425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.404367 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.404425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.404390 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.404425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.404408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02d58084-9a2b-49c6-a79e-0d09c632d179-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.404647 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.404526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-tls\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.404733 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.404641 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c466071-335d-4f93-92d8-2830b8f01b18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.405495 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.405477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7514196b-f24b-4e94-a455-379adc691f79-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.416923 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.416900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qql6\" (UniqueName: \"kubernetes.io/projected/6c466071-335d-4f93-92d8-2830b8f01b18-kube-api-access-8qql6\") pod \"node-exporter-2zjlf\" (UID: \"6c466071-335d-4f93-92d8-2830b8f01b18\") " pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.417022 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.416985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lljw\" (UniqueName: \"kubernetes.io/projected/7514196b-f24b-4e94-a455-379adc691f79-kube-api-access-2lljw\") pod \"kube-state-metrics-74cb57df7-9mdjs\" (UID: \"7514196b-f24b-4e94-a455-379adc691f79\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.417423 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.417405 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99m7\" (UniqueName: \"kubernetes.io/projected/02d58084-9a2b-49c6-a79e-0d09c632d179-kube-api-access-n99m7\") pod \"openshift-state-metrics-657fd6c54-2p9fk\" (UID: \"02d58084-9a2b-49c6-a79e-0d09c632d179\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.503947 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.503863 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" Sep 08 16:52:11.528083 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.528050 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2zjlf" Sep 08 16:52:11.533197 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.533063 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" Sep 08 16:52:11.538488 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:11.538457 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c466071_335d_4f93_92d8_2830b8f01b18.slice/crio-4d8a21f4c4b2e549f1047d235e8699673a974644222ac3ccca3ebf18e0452dad WatchSource:0}: Error finding container 4d8a21f4c4b2e549f1047d235e8699673a974644222ac3ccca3ebf18e0452dad: Status 404 returned error can't find the container with id 4d8a21f4c4b2e549f1047d235e8699673a974644222ac3ccca3ebf18e0452dad Sep 08 16:52:11.617187 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.617155 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2zjlf" event={"ID":"6c466071-335d-4f93-92d8-2830b8f01b18","Type":"ContainerStarted","Data":"4d8a21f4c4b2e549f1047d235e8699673a974644222ac3ccca3ebf18e0452dad"} Sep 08 16:52:11.777602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.777530 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk"] Sep 08 16:52:11.780381 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:11.780338 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d58084_9a2b_49c6_a79e_0d09c632d179.slice/crio-1935624b18ba436bb99f52a12907d0fe3dde505ef9ce4c6e90004fcf7eb67c60 WatchSource:0}: Error finding container 1935624b18ba436bb99f52a12907d0fe3dde505ef9ce4c6e90004fcf7eb67c60: Status 404 returned error can't find the container with id 1935624b18ba436bb99f52a12907d0fe3dde505ef9ce4c6e90004fcf7eb67c60 Sep 08 16:52:11.811776 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:11.811756 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs"] Sep 08 16:52:11.814336 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:11.814300 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7514196b_f24b_4e94_a455_379adc691f79.slice/crio-b7a7e71bc1da3cee685697e3d0e0aad3bb705ee427ebef541c0b716350bc3e30 WatchSource:0}: Error finding container b7a7e71bc1da3cee685697e3d0e0aad3bb705ee427ebef541c0b716350bc3e30: Status 404 returned error can't find the container with id b7a7e71bc1da3cee685697e3d0e0aad3bb705ee427ebef541c0b716350bc3e30 Sep 08 16:52:12.289016 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.288984 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:52:12.291711 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.291655 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.295197 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295171 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 16:52:12.295412 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295383 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-524nd\"" Sep 08 16:52:12.295536 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295518 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 16:52:12.295617 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295543 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 16:52:12.295722 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295709 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 16:52:12.295836 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295779 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 16:52:12.295902 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.295865 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 16:52:12.296045 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.296030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 16:52:12.297300 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.297254 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 16:52:12.299755 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.299549 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 16:52:12.309582 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.309542 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:52:12.413377 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413339 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413548 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413548 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413445 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413548 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413548 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413525 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lpv\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413579 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.413764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413759 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.414070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.414070 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.413904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514671 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514637 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514864 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514773 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514864 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514826 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514864 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514932 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.514980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lpv\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.514986 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515014 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515041 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515461 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515170 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.515461 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.515388 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.516413 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:52:12.515850 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle podName:1231a9ff-3fd0-4b18-b006-815f0ebfe07a nodeName:}" failed. No retries permitted until 2026-09-08 16:52:13.015822949 +0000 UTC m=+64.160593490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a") : configmap references non-existent config key: ca-bundle.crt Sep 08 16:52:12.517313 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.517284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.518391 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.518344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.519908 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.519879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.520403 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.520361 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.520633 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.520607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.521034 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.521010 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.521492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.521473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.521576 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.521521 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.521914 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.521890 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.522098 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.522077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.524228 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.524189 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lpv\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:12.623174 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.623139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" event={"ID":"7514196b-f24b-4e94-a455-379adc691f79","Type":"ContainerStarted","Data":"b7a7e71bc1da3cee685697e3d0e0aad3bb705ee427ebef541c0b716350bc3e30"} Sep 08 16:52:12.625952 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.625919 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" event={"ID":"02d58084-9a2b-49c6-a79e-0d09c632d179","Type":"ContainerStarted","Data":"3ac586962eaa696c4e9a449f1f800d15091003bd019c6c1a035f4b66da9dcdb6"} Sep 08 16:52:12.626093 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.625984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" event={"ID":"02d58084-9a2b-49c6-a79e-0d09c632d179","Type":"ContainerStarted","Data":"97ec5c964f374b7ffe0e27d19c859877ea0b86497cc24475ab404e616dddd54f"} Sep 08 16:52:12.626093 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:12.626013 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" event={"ID":"02d58084-9a2b-49c6-a79e-0d09c632d179","Type":"ContainerStarted","Data":"1935624b18ba436bb99f52a12907d0fe3dde505ef9ce4c6e90004fcf7eb67c60"} Sep 08 16:52:13.020764 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.020727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:13.021682 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.021658 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:13.204103 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.204080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:52:13.228880 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.228854 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59f5fb6d78-6jmft"] Sep 08 16:52:13.233794 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.233771 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.237064 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.236841 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 08 16:52:13.237064 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.236935 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 08 16:52:13.237265 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.237248 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-56mbvi58770kl\"" Sep 08 16:52:13.238268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.238038 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 08 16:52:13.238268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.238093 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 08 16:52:13.238268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.238111 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 08 16:52:13.238268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.238045 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4vrgc\"" Sep 08 16:52:13.249740 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.249656 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59f5fb6d78-6jmft"] Sep 08 16:52:13.324184 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324144 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324277 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4abbe321-dac0-433b-b533-1b5afa2cc179-metrics-client-ca\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-grpc-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324405 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324458 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp27l\" (UniqueName: \"kubernetes.io/projected/4abbe321-dac0-433b-b533-1b5afa2cc179-kube-api-access-bp27l\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.324629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.324493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425124 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425084 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425264 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425138 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425264 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp27l\" (UniqueName: \"kubernetes.io/projected/4abbe321-dac0-433b-b533-1b5afa2cc179-kube-api-access-bp27l\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425264 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425399 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425273 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425399 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425342 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425399 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4abbe321-dac0-433b-b533-1b5afa2cc179-metrics-client-ca\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.425485 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.425408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-grpc-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.426553 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.426483 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4abbe321-dac0-433b-b533-1b5afa2cc179-metrics-client-ca\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.428874 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.428786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-grpc-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.428993 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.428966 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.429193 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.429170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.429362 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.429340 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.429658 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.429607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.430180 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.430151 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4abbe321-dac0-433b-b533-1b5afa2cc179-secret-thanos-querier-tls\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.434867 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.434809 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp27l\" (UniqueName: \"kubernetes.io/projected/4abbe321-dac0-433b-b533-1b5afa2cc179-kube-api-access-bp27l\") pod \"thanos-querier-59f5fb6d78-6jmft\" (UID: \"4abbe321-dac0-433b-b533-1b5afa2cc179\") " pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.502721 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.502665 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:52:13.506587 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:13.506555 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1231a9ff_3fd0_4b18_b006_815f0ebfe07a.slice/crio-8ab825e7672c2459ea650577f3aa0308fe569854899b614bb2a46c72d9c1da40 WatchSource:0}: Error finding container 8ab825e7672c2459ea650577f3aa0308fe569854899b614bb2a46c72d9c1da40: Status 404 returned error can't find the container with id 8ab825e7672c2459ea650577f3aa0308fe569854899b614bb2a46c72d9c1da40 Sep 08 16:52:13.561041 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.561010 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:13.631597 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.631563 2556 generic.go:356] "Generic (PLEG): container finished" podID="6c466071-335d-4f93-92d8-2830b8f01b18" containerID="c51caab8426cbf26ae772fa457eb1e448006db8d6452493a0703659ff86b833f" exitCode=0 Sep 08 16:52:13.631759 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.631653 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2zjlf" event={"ID":"6c466071-335d-4f93-92d8-2830b8f01b18","Type":"ContainerDied","Data":"c51caab8426cbf26ae772fa457eb1e448006db8d6452493a0703659ff86b833f"} Sep 08 16:52:13.634509 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.634481 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" event={"ID":"7514196b-f24b-4e94-a455-379adc691f79","Type":"ContainerStarted","Data":"cf562e1b9fdc36981127840bd9bb21c96c225f935cd9f1776b86ca8d63d59ed7"} Sep 08 16:52:13.634622 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.634539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" event={"ID":"7514196b-f24b-4e94-a455-379adc691f79","Type":"ContainerStarted","Data":"ccb767988f57ba327eb98c95882a0c2247dc5404fd3402a7c136441b036d3bc3"} Sep 08 16:52:13.634622 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.634566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" event={"ID":"7514196b-f24b-4e94-a455-379adc691f79","Type":"ContainerStarted","Data":"f6fd9b359220a8fec757c1612953b9908fab7995f7b0af9f164e228381d60dd1"} Sep 08 16:52:13.636261 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.636240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"8ab825e7672c2459ea650577f3aa0308fe569854899b614bb2a46c72d9c1da40"} Sep 08 16:52:13.671979 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:13.671924 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-9mdjs" podStartSLOduration=1.33510492 podStartE2EDuration="2.671901757s" totalImagesPullingTime="1.336796837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:13.670980231 +0000 UTC m=+64.815750793" watchObservedRunningTime="2026-09-08 16:52:13.671901757 +0000 UTC m=+64.816672319" Sep 08 16:52:14.037166 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.037091 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59f5fb6d78-6jmft"] Sep 08 16:52:14.039904 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:14.039877 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abbe321_dac0_433b_b533_1b5afa2cc179.slice/crio-a94527b1e8a7cec51a561dc0c69e58938748eb15a6b39d952fc6ecceb5d58b85 WatchSource:0}: Error finding container a94527b1e8a7cec51a561dc0c69e58938748eb15a6b39d952fc6ecceb5d58b85: Status 404 returned error can't find the container with id a94527b1e8a7cec51a561dc0c69e58938748eb15a6b39d952fc6ecceb5d58b85 Sep 08 16:52:14.132970 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.132932 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:14.133148 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.133005 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:14.136241 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.136206 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 16:52:14.136241 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.136232 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 16:52:14.146117 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.146086 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9853d638-cf47-4f8f-b45f-0a83f02af5d1-original-pull-secret\") pod \"global-pull-secret-syncer-zmtz9\" (UID: \"9853d638-cf47-4f8f-b45f-0a83f02af5d1\") " pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:14.146262 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.146094 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5acd11f2-9934-413b-ac7a-4646902745f6-metrics-certs\") pod \"network-metrics-daemon-j74bn\" (UID: \"5acd11f2-9934-413b-ac7a-4646902745f6\") " pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:14.233849 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.233808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:14.236871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.236844 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 16:52:14.246633 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.246610 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 16:52:14.257448 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.257420 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjr7\" (UniqueName: \"kubernetes.io/projected/7dd46390-5e12-4101-9a78-8a3444ed9d87-kube-api-access-wjjr7\") pod \"network-check-target-sdrwc\" (UID: \"7dd46390-5e12-4101-9a78-8a3444ed9d87\") " pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:14.369935 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.369906 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8582r\"" Sep 08 16:52:14.374141 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.374115 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmtz9" Sep 08 16:52:14.377297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.377275 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:14.382660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.382639 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\"" Sep 08 16:52:14.389785 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.389760 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j74bn" Sep 08 16:52:14.644526 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.644490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2zjlf" event={"ID":"6c466071-335d-4f93-92d8-2830b8f01b18","Type":"ContainerStarted","Data":"175855bc9c49b237a347dc5b509ce8e1c4bed569eac00ad1cfc6fe8038e91640"} Sep 08 16:52:14.644645 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.644561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2zjlf" event={"ID":"6c466071-335d-4f93-92d8-2830b8f01b18","Type":"ContainerStarted","Data":"f2652affabae0144ab917ade4fb944dd42a4a8c803e66fdd4c3f069a4d4aab28"} Sep 08 16:52:14.647373 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.647339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" event={"ID":"02d58084-9a2b-49c6-a79e-0d09c632d179","Type":"ContainerStarted","Data":"72caa5ee3669630aae538deca5a16cf35cb88c6c7cb9819f496d8715855a5164"} Sep 08 16:52:14.652875 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.652844 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"a94527b1e8a7cec51a561dc0c69e58938748eb15a6b39d952fc6ecceb5d58b85"} Sep 08 16:52:14.668920 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.668869 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2zjlf" podStartSLOduration=2.3589109600000002 podStartE2EDuration="3.668851083s" totalImagesPullingTime="1.240682976s" totalInitContainerRuntime="69.257147ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:14.667317086 +0000 UTC m=+65.812087655" watchObservedRunningTime="2026-09-08 16:52:14.668851083 +0000 UTC m=+65.813621640" Sep 08 16:52:14.690522 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.690469 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2p9fk" podStartSLOduration=1.756121202 podStartE2EDuration="3.690443989s" totalImagesPullingTime="1.934322787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:14.687803201 +0000 UTC m=+65.832573763" watchObservedRunningTime="2026-09-08 16:52:14.690443989 +0000 UTC m=+65.835214558" Sep 08 16:52:14.913807 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.913729 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmtz9"] Sep 08 16:52:14.916984 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:14.916957 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9853d638_cf47_4f8f_b45f_0a83f02af5d1.slice/crio-2ea1180b14d1725a1b235b00c728140e1cf60b94f98fa71432b602d28b7da8ec WatchSource:0}: Error finding container 2ea1180b14d1725a1b235b00c728140e1cf60b94f98fa71432b602d28b7da8ec: Status 404 returned error can't find the container with id 2ea1180b14d1725a1b235b00c728140e1cf60b94f98fa71432b602d28b7da8ec Sep 08 16:52:14.921355 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.921305 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j74bn"] Sep 08 16:52:14.923651 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:14.923625 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acd11f2_9934_413b_ac7a_4646902745f6.slice/crio-02fb9d070baa07cc0ab43eb9a81b93c07ba1dbc4285aa7ce71e1b6bcfba16f45 WatchSource:0}: Error finding container 02fb9d070baa07cc0ab43eb9a81b93c07ba1dbc4285aa7ce71e1b6bcfba16f45: Status 404 returned error can't find the container with id 02fb9d070baa07cc0ab43eb9a81b93c07ba1dbc4285aa7ce71e1b6bcfba16f45 Sep 08 16:52:14.925582 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:14.925560 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sdrwc"] Sep 08 16:52:14.927861 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:14.927815 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd46390_5e12_4101_9a78_8a3444ed9d87.slice/crio-54d40a590b270ca4b2b6d6482d6f5549271753b0dc5289115863809a581de356 WatchSource:0}: Error finding container 54d40a590b270ca4b2b6d6482d6f5549271753b0dc5289115863809a581de356: Status 404 returned error can't find the container with id 54d40a590b270ca4b2b6d6482d6f5549271753b0dc5289115863809a581de356 Sep 08 16:52:15.659711 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.659674 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdrwc" event={"ID":"7dd46390-5e12-4101-9a78-8a3444ed9d87","Type":"ContainerStarted","Data":"54d40a590b270ca4b2b6d6482d6f5549271753b0dc5289115863809a581de356"} Sep 08 16:52:15.661616 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.661587 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="e0bcdcd3c9d9c853a7833059d5157c9904130f6c721a80c506335559ffddbd94" exitCode=0 Sep 08 16:52:15.661720 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.661664 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"e0bcdcd3c9d9c853a7833059d5157c9904130f6c721a80c506335559ffddbd94"} Sep 08 16:52:15.664469 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.664426 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmtz9" event={"ID":"9853d638-cf47-4f8f-b45f-0a83f02af5d1","Type":"ContainerStarted","Data":"2ea1180b14d1725a1b235b00c728140e1cf60b94f98fa71432b602d28b7da8ec"} Sep 08 16:52:15.667106 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.666705 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j74bn" event={"ID":"5acd11f2-9934-413b-ac7a-4646902745f6","Type":"ContainerStarted","Data":"02fb9d070baa07cc0ab43eb9a81b93c07ba1dbc4285aa7ce71e1b6bcfba16f45"} Sep 08 16:52:15.700386 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.700350 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bf45677b9-6lzw4"] Sep 08 16:52:15.704181 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.704159 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.718580 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.718558 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 08 16:52:15.718694 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.718677 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 08 16:52:15.719007 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.718982 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-19bi2bt3ghkc9\"" Sep 08 16:52:15.720114 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.719089 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 08 16:52:15.720114 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.719201 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 08 16:52:15.720114 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.719088 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-plllv\"" Sep 08 16:52:15.735471 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.735440 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bf45677b9-6lzw4"] Sep 08 16:52:15.750145 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/77dc8729-7ba0-4229-8dce-2c61a315c4ff-audit-log\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750272 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750178 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-metrics-server-audit-profiles\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750479 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750456 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-client-certs\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750770 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750742 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqgh\" (UniqueName: \"kubernetes.io/projected/77dc8729-7ba0-4229-8dce-2c61a315c4ff-kube-api-access-tnqgh\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750847 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750814 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-tls\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750904 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.750960 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.750935 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-client-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852493 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852443 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-tls\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852676 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852504 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852676 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852542 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-client-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852676 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852577 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/77dc8729-7ba0-4229-8dce-2c61a315c4ff-audit-log\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852676 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-metrics-server-audit-profiles\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852676 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852669 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-client-certs\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.852932 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.852752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqgh\" (UniqueName: \"kubernetes.io/projected/77dc8729-7ba0-4229-8dce-2c61a315c4ff-kube-api-access-tnqgh\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.853726 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.853679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.854777 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.854752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/77dc8729-7ba0-4229-8dce-2c61a315c4ff-metrics-server-audit-profiles\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.855030 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.855005 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/77dc8729-7ba0-4229-8dce-2c61a315c4ff-audit-log\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.855678 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.855653 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-tls\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.858078 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.858034 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-client-ca-bundle\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.858754 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.858731 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/77dc8729-7ba0-4229-8dce-2c61a315c4ff-secret-metrics-server-client-certs\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.862574 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.862551 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqgh\" (UniqueName: \"kubernetes.io/projected/77dc8729-7ba0-4229-8dce-2c61a315c4ff-kube-api-access-tnqgh\") pod \"metrics-server-6bf45677b9-6lzw4\" (UID: \"77dc8729-7ba0-4229-8dce-2c61a315c4ff\") " pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:15.983903 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.982608 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t"] Sep 08 16:52:15.987309 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.987282 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:15.990989 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.990963 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kt8nn\"" Sep 08 16:52:15.991161 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.991145 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 08 16:52:15.999432 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:15.999380 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t"] Sep 08 16:52:16.017663 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.017635 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:16.054163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.054131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.054358 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.054219 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vks\" (UniqueName: \"kubernetes.io/projected/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-kube-api-access-w4vks\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.154688 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.154651 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.154848 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.154736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vks\" (UniqueName: \"kubernetes.io/projected/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-kube-api-access-w4vks\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.157774 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.157745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.163308 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.163282 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vks\" (UniqueName: \"kubernetes.io/projected/3803fb5f-cd1a-42e9-8c3f-87e2b76f4844-kube-api-access-w4vks\") pod \"monitoring-plugin-85bbbbbf4-sp77t\" (UID: \"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.302635 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.302367 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:16.452251 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.450892 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-799f5c6974-z9tbx"] Sep 08 16:52:16.455441 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.454595 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.460971 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.460950 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-84rx2\"" Sep 08 16:52:16.461267 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.461081 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 08 16:52:16.462114 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.461573 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 08 16:52:16.462114 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.461623 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 08 16:52:16.463219 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.462459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 08 16:52:16.463219 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.462674 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 08 16:52:16.480605 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.480575 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 08 16:52:16.490596 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.490569 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-799f5c6974-z9tbx"] Sep 08 16:52:16.558684 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558598 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-federate-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.558684 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558653 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-metrics-client-ca\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.558922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6df\" (UniqueName: \"kubernetes.io/projected/024b00b1-9b61-4fbd-888b-5b70d1980e60-kube-api-access-kf6df\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.558922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558795 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-serving-certs-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.558922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.558922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-trusted-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.559110 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558932 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.559110 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.558961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660385 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660446 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-federate-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660483 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-metrics-client-ca\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6df\" (UniqueName: \"kubernetes.io/projected/024b00b1-9b61-4fbd-888b-5b70d1980e60-kube-api-access-kf6df\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-serving-certs-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660604 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.661608 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.660630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-trusted-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.662344 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.661823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-trusted-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.662344 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.662112 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-metrics-client-ca\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.662866 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.662738 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024b00b1-9b61-4fbd-888b-5b70d1980e60-serving-certs-ca-bundle\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.671456 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.668637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.671456 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.670569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-secret-telemeter-client\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.671456 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.670938 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-telemeter-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.672010 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.671968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/024b00b1-9b61-4fbd-888b-5b70d1980e60-federate-client-tls\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.673611 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.673587 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6df\" (UniqueName: \"kubernetes.io/projected/024b00b1-9b61-4fbd-888b-5b70d1980e60-kube-api-access-kf6df\") pod \"telemeter-client-799f5c6974-z9tbx\" (UID: \"024b00b1-9b61-4fbd-888b-5b70d1980e60\") " pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.783145 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.783122 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" Sep 08 16:52:16.917573 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.917533 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t"] Sep 08 16:52:16.921665 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:16.921482 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bf45677b9-6lzw4"] Sep 08 16:52:17.087846 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.086725 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-799f5c6974-z9tbx"] Sep 08 16:52:17.091049 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:52:17.090972 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024b00b1_9b61_4fbd_888b_5b70d1980e60.slice/crio-8e0bd78a085965ae63c1c2a2e286b530c7ab968ddff42dbe1a96eb14ff49ecfa WatchSource:0}: Error finding container 8e0bd78a085965ae63c1c2a2e286b530c7ab968ddff42dbe1a96eb14ff49ecfa: Status 404 returned error can't find the container with id 8e0bd78a085965ae63c1c2a2e286b530c7ab968ddff42dbe1a96eb14ff49ecfa Sep 08 16:52:17.581671 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.579365 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:52:17.586042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.586018 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.606951 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.606212 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 08 16:52:17.606951 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.606544 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 08 16:52:17.606951 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.606745 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 08 16:52:17.607497 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.607300 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 08 16:52:17.608803 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.608661 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 08 16:52:17.609034 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609008 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7lnj6\"" Sep 08 16:52:17.609575 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609197 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 08 16:52:17.609575 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609293 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 08 16:52:17.609575 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609420 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 08 16:52:17.609575 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609512 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 08 16:52:17.610422 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.609989 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 08 16:52:17.610422 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.610235 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 08 16:52:17.622297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.610799 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vmlmlj5d1ul0\"" Sep 08 16:52:17.622297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.611135 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 08 16:52:17.628671 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.625923 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 08 16:52:17.628671 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.627132 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 08 16:52:17.630152 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.630133 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677794 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677887 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677961 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.677997 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678029 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678100 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678218 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678247 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678412 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678471 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.682342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678551 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.683473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.683473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678682 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bds\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.683473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678766 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.683473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.678853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.704433 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.704294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" event={"ID":"77dc8729-7ba0-4229-8dce-2c61a315c4ff","Type":"ContainerStarted","Data":"236c12f815bc850adf5e135fa59ef6bcea362c3a76ba048a25cae7db50004f39"} Sep 08 16:52:17.712286 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.712250 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"17632f831cacafd2c192b9958c689c8ea8327f6ecf123326013dffe80b29bccb"} Sep 08 16:52:17.712478 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.712345 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"b61db8c8ae990cec5472fb0653e624da4adb58e43ca8010ea8ac6b8884132374"} Sep 08 16:52:17.712478 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.712384 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"d410c46d90b96cf0f1d0d6e15cb99f13beb598b51c7e9661620c906660f9242b"} Sep 08 16:52:17.716677 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.716647 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j74bn" event={"ID":"5acd11f2-9934-413b-ac7a-4646902745f6","Type":"ContainerStarted","Data":"ec53e83934c5eee7c7841be99259219f958d05e9a8c08217487fe854f17d075b"} Sep 08 16:52:17.716814 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.716713 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j74bn" event={"ID":"5acd11f2-9934-413b-ac7a-4646902745f6","Type":"ContainerStarted","Data":"b5bf1fc3dca28c08d5d986b84e02a09535be8a09a2435f5b19f5ad77ace2b64c"} Sep 08 16:52:17.723673 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.723262 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" event={"ID":"024b00b1-9b61-4fbd-888b-5b70d1980e60","Type":"ContainerStarted","Data":"8e0bd78a085965ae63c1c2a2e286b530c7ab968ddff42dbe1a96eb14ff49ecfa"} Sep 08 16:52:17.729083 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.729017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" event={"ID":"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844","Type":"ContainerStarted","Data":"a911441a0c62828f216a2492271350537809201303affe2024faff75ef66defc"} Sep 08 16:52:17.739402 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.738731 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j74bn" podStartSLOduration=66.030309454 podStartE2EDuration="1m7.738713075s" totalImagesPullingTime="1.708403621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:17.738313363 +0000 UTC m=+68.883083948" watchObservedRunningTime="2026-09-08 16:52:17.738713075 +0000 UTC m=+68.883483638" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.779885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.779947 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.779981 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780250 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780309 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780396 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bds\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.781405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.782364 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780566 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.782364 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.782364 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.780622 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.785679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.783455 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.785679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.783508 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.785679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.783728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.785679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.784149 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.785679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.784551 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.788465 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.786599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.788465 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.787988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.791033 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.790572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.791378 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.791130 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.792661 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.792482 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.793395 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.793225 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.796885 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.796860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bds\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.800474 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.800381 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.800474 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.800447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.800980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.800958 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.801439 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.801416 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.802042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.801817 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.802042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.801990 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.803945 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.803906 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.810504 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.810463 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config\") pod \"prometheus-k8s-0\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:17.906150 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:17.906104 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:19.615519 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:19.615485 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xc9ml" Sep 08 16:52:23.398122 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.398010 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:52:23.755815 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.755780 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sdrwc" event={"ID":"7dd46390-5e12-4101-9a78-8a3444ed9d87","Type":"ContainerStarted","Data":"f34a2a0569f54f3a73533cddc439b3b6f605a5c0ebde9f7b8f26ff24e7f9c85c"} Sep 08 16:52:23.755974 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.755936 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:23.758887 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.758862 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"811a06c5f6bdd168642cdef73174c4c93251a2623d5c8d609dbe41ad4e84e0f6"} Sep 08 16:52:23.759018 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.758922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"3533d8878398cf3d0c2e1b1918e35554d9c96f1d8c64eb6c94b68285aeec8118"} Sep 08 16:52:23.759018 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.758954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"adaa71966f8cc09d896740f6cbf9e5c6914b5ff9c5ecc5dfb39c4e3a43aaa619"} Sep 08 16:52:23.759018 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.758995 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"ecfdd07cf0d3e87ffafe7aafb32d6b432ffe87a6fa290803bd053eb28060b4a2"} Sep 08 16:52:23.759154 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.759023 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"e8dfa62471fc69e61cdd1f6b4e91f41742b8b72d845058b51f16e519ecad48f0"} Sep 08 16:52:23.759154 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.759050 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerStarted","Data":"6819ec89697cea3764082ef80a75a5545ca770de640fac434dfa9141e19d0a86"} Sep 08 16:52:23.760453 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.760430 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmtz9" event={"ID":"9853d638-cf47-4f8f-b45f-0a83f02af5d1","Type":"ContainerStarted","Data":"0e694d8a52236b8ead62eca85b8c508dc496aa71c7fd235bd3e8c9496acfd899"} Sep 08 16:52:23.761812 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.761791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" event={"ID":"77dc8729-7ba0-4229-8dce-2c61a315c4ff","Type":"ContainerStarted","Data":"0207cdd3e296ff594678e9c41e5dea1c67da3d19a5d3c2b577ef3ef188d55150"} Sep 08 16:52:23.764810 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.764788 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"7d8d23ed319b928f27ca6216c6054de457d23d764b992a3f392a126ee0f5bc1e"} Sep 08 16:52:23.764917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.764842 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"9c448b51275f8ac1748caf77d66772dc05e06a35a341e0f37b0623e061b72ee5"} Sep 08 16:52:23.764917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.764872 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" event={"ID":"4abbe321-dac0-433b-b533-1b5afa2cc179","Type":"ContainerStarted","Data":"33345b816249b41f90286e870b8fd644a09c1e8cb4caa186cc3ecaeedd184d89"} Sep 08 16:52:23.764917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.764916 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:23.767086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.767060 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" event={"ID":"024b00b1-9b61-4fbd-888b-5b70d1980e60","Type":"ContainerStarted","Data":"bdba2a47522224cd82a31f9e2193ef4399fba0050e4ff428d873665a635e1fae"} Sep 08 16:52:23.767176 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.767121 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" event={"ID":"024b00b1-9b61-4fbd-888b-5b70d1980e60","Type":"ContainerStarted","Data":"c030689ae04141acef85a470eda02193b04bc0e42407dde81fb3d17b620b81f5"} Sep 08 16:52:23.767176 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.767151 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" event={"ID":"024b00b1-9b61-4fbd-888b-5b70d1980e60","Type":"ContainerStarted","Data":"834d96fed654c72e779acf1e7a6927cfeb233d6ca1babe3b712967651888e191"} Sep 08 16:52:23.768578 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.768549 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" event={"ID":"3803fb5f-cd1a-42e9-8c3f-87e2b76f4844","Type":"ContainerStarted","Data":"be6a69514c4897ff9ccc7d8c79e3089f43c96d5d7c4540f2f7a86bee274deceb"} Sep 08 16:52:23.768758 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.768736 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:23.770311 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.770287 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" exitCode=0 Sep 08 16:52:23.770440 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.770341 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} Sep 08 16:52:23.770440 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.770396 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"d91af50ef63631dce208bf4c206ea442d3eae8ba17046fd23641ec10341c58f4"} Sep 08 16:52:23.771404 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.771386 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" Sep 08 16:52:23.774158 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.774143 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" Sep 08 16:52:23.829710 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.829601 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" podStartSLOduration=2.673244766 podStartE2EDuration="8.829572417s" totalImagesPullingTime="6.156327651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.829006988 +0000 UTC m=+74.973777559" watchObservedRunningTime="2026-09-08 16:52:23.829572417 +0000 UTC m=+74.974342978" Sep 08 16:52:23.830514 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.830476 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sdrwc" podStartSLOduration=65.664319273 podStartE2EDuration="1m13.830465118s" totalImagesPullingTime="8.166145845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.784466827 +0000 UTC m=+74.929237401" watchObservedRunningTime="2026-09-08 16:52:23.830465118 +0000 UTC m=+74.975235681" Sep 08 16:52:23.863884 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.863834 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-799f5c6974-z9tbx" podStartSLOduration=1.862799811 podStartE2EDuration="7.863818162s" totalImagesPullingTime="6.001018351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.86067161 +0000 UTC m=+75.005442170" watchObservedRunningTime="2026-09-08 16:52:23.863818162 +0000 UTC m=+75.008588723" Sep 08 16:52:23.914529 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.914430 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.269957285 podStartE2EDuration="11.914411686s" totalImagesPullingTime="8.627396416s" totalInitContainerRuntime="17.057985ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.911919665 +0000 UTC m=+75.056690219" watchObservedRunningTime="2026-09-08 16:52:23.914411686 +0000 UTC m=+75.059182246" Sep 08 16:52:23.955084 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.955015 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59f5fb6d78-6jmft" podStartSLOduration=2.224101185 podStartE2EDuration="10.954993777s" totalImagesPullingTime="8.730892592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.952337794 +0000 UTC m=+75.097108347" watchObservedRunningTime="2026-09-08 16:52:23.954993777 +0000 UTC m=+75.099764339" Sep 08 16:52:23.982459 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:23.982402 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-sp77t" podStartSLOduration=2.819906726 podStartE2EDuration="8.982383909s" totalImagesPullingTime="6.162477183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:23.981138228 +0000 UTC m=+75.125908783" watchObservedRunningTime="2026-09-08 16:52:23.982383909 +0000 UTC m=+75.127154488" Sep 08 16:52:27.798815 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.798777 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} Sep 08 16:52:27.799167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.798859 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} Sep 08 16:52:27.799167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.798900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} Sep 08 16:52:27.799167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.798935 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} Sep 08 16:52:27.799167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.798971 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} Sep 08 16:52:27.799167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.799008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerStarted","Data":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} Sep 08 16:52:27.840661 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.840606 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.703741141 podStartE2EDuration="10.840587285s" totalImagesPullingTime="3.112369812s" totalInitContainerRuntime="24.476332ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:27.838740712 +0000 UTC m=+78.983511273" watchObservedRunningTime="2026-09-08 16:52:27.840587285 +0000 UTC m=+78.985357845" Sep 08 16:52:27.840819 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.840794 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zmtz9" podStartSLOduration=69.670080646 podStartE2EDuration="1m17.840786727s" totalImagesPullingTime="8.170706081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:24.060287386 +0000 UTC m=+75.205057946" watchObservedRunningTime="2026-09-08 16:52:27.840786727 +0000 UTC m=+78.985557288" Sep 08 16:52:27.907387 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:27.907358 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:52:36.018411 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:36.018283 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:36.018411 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:36.018350 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:54.780813 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:54.780776 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sdrwc" Sep 08 16:52:56.023493 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:56.023462 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:52:56.027223 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:52:56.027203 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bf45677b9-6lzw4" Sep 08 16:53:17.906633 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:17.906587 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:17.925941 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:17.925918 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:18.018810 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:18.018784 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:31.860120 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.860079 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:31.860712 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.860572 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="alertmanager" containerID="cri-o://6819ec89697cea3764082ef80a75a5545ca770de640fac434dfa9141e19d0a86" gracePeriod=120 Sep 08 16:53:31.860900 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.860866 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="prom-label-proxy" containerID="cri-o://811a06c5f6bdd168642cdef73174c4c93251a2623d5c8d609dbe41ad4e84e0f6" gracePeriod=120 Sep 08 16:53:31.861000 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.860869 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-metric" containerID="cri-o://3533d8878398cf3d0c2e1b1918e35554d9c96f1d8c64eb6c94b68285aeec8118" gracePeriod=120 Sep 08 16:53:31.861050 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.861020 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-web" containerID="cri-o://ecfdd07cf0d3e87ffafe7aafb32d6b432ffe87a6fa290803bd053eb28060b4a2" gracePeriod=120 Sep 08 16:53:31.861139 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.861115 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy" containerID="cri-o://adaa71966f8cc09d896740f6cbf9e5c6914b5ff9c5ecc5dfb39c4e3a43aaa619" gracePeriod=120 Sep 08 16:53:31.861213 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:31.861164 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="config-reloader" containerID="cri-o://e8dfa62471fc69e61cdd1f6b4e91f41742b8b72d845058b51f16e519ecad48f0" gracePeriod=120 Sep 08 16:53:32.059130 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059097 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="811a06c5f6bdd168642cdef73174c4c93251a2623d5c8d609dbe41ad4e84e0f6" exitCode=0 Sep 08 16:53:32.059130 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059122 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="3533d8878398cf3d0c2e1b1918e35554d9c96f1d8c64eb6c94b68285aeec8118" exitCode=0 Sep 08 16:53:32.059130 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059130 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="adaa71966f8cc09d896740f6cbf9e5c6914b5ff9c5ecc5dfb39c4e3a43aaa619" exitCode=0 Sep 08 16:53:32.059130 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059136 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="e8dfa62471fc69e61cdd1f6b4e91f41742b8b72d845058b51f16e519ecad48f0" exitCode=0 Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059143 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="6819ec89697cea3764082ef80a75a5545ca770de640fac434dfa9141e19d0a86" exitCode=0 Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059164 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"811a06c5f6bdd168642cdef73174c4c93251a2623d5c8d609dbe41ad4e84e0f6"} Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059249 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"3533d8878398cf3d0c2e1b1918e35554d9c96f1d8c64eb6c94b68285aeec8118"} Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"adaa71966f8cc09d896740f6cbf9e5c6914b5ff9c5ecc5dfb39c4e3a43aaa619"} Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059299 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"e8dfa62471fc69e61cdd1f6b4e91f41742b8b72d845058b51f16e519ecad48f0"} Sep 08 16:53:32.059389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:32.059337 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"6819ec89697cea3764082ef80a75a5545ca770de640fac434dfa9141e19d0a86"} Sep 08 16:53:33.067075 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.067046 2556 generic.go:356] "Generic (PLEG): container finished" podID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerID="ecfdd07cf0d3e87ffafe7aafb32d6b432ffe87a6fa290803bd053eb28060b4a2" exitCode=0 Sep 08 16:53:33.067415 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.067102 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"ecfdd07cf0d3e87ffafe7aafb32d6b432ffe87a6fa290803bd053eb28060b4a2"} Sep 08 16:53:33.108202 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.108180 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:33.151788 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151757 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.151915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151794 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.151915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151823 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.151915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151850 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.151915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151878 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.151915 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151914 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151945 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.151973 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54lpv\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152004 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152052 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152100 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152138 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152501 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152202 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls\") pod \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\" (UID: \"1231a9ff-3fd0-4b18-b006-815f0ebfe07a\") " Sep 08 16:53:33.152501 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152404 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:53:33.152593 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152569 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-main-db\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.152966 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152645 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:33.152966 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.152699 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:33.158010 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.157940 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.158010 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.157942 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:53:33.158192 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.158022 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.160687 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.160573 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.160687 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.160659 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv" (OuterVolumeSpecName: "kube-api-access-54lpv") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "kube-api-access-54lpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:53:33.160878 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.160712 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out" (OuterVolumeSpecName: "config-out") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:53:33.161091 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.161056 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume" (OuterVolumeSpecName: "config-volume") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.161230 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.161209 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.163733 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.163705 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.172861 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.172835 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config" (OuterVolumeSpecName: "web-config") pod "1231a9ff-3fd0-4b18-b006-815f0ebfe07a" (UID: "1231a9ff-3fd0-4b18-b006-815f0ebfe07a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253114 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253140 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lpv\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-kube-api-access-54lpv\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253151 2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253159 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253168 2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-cluster-tls-config\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253177 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-config-out\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253186 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-main-tls\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253190 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253194 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253542 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253203 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253542 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253212 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-metrics-client-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253542 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253220 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-web-config\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:33.253542 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:33.253227 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1231a9ff-3fd0-4b18-b006-815f0ebfe07a-tls-assets\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:34.074228 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.074192 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1231a9ff-3fd0-4b18-b006-815f0ebfe07a","Type":"ContainerDied","Data":"8ab825e7672c2459ea650577f3aa0308fe569854899b614bb2a46c72d9c1da40"} Sep 08 16:53:34.074228 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.074219 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.074677 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.074274 2556 scope.go:117] "RemoveContainer" containerID="811a06c5f6bdd168642cdef73174c4c93251a2623d5c8d609dbe41ad4e84e0f6" Sep 08 16:53:34.083136 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.083113 2556 scope.go:117] "RemoveContainer" containerID="3533d8878398cf3d0c2e1b1918e35554d9c96f1d8c64eb6c94b68285aeec8118" Sep 08 16:53:34.090512 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.090497 2556 scope.go:117] "RemoveContainer" containerID="adaa71966f8cc09d896740f6cbf9e5c6914b5ff9c5ecc5dfb39c4e3a43aaa619" Sep 08 16:53:34.096959 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.096945 2556 scope.go:117] "RemoveContainer" containerID="ecfdd07cf0d3e87ffafe7aafb32d6b432ffe87a6fa290803bd053eb28060b4a2" Sep 08 16:53:34.103346 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.103310 2556 scope.go:117] "RemoveContainer" containerID="e8dfa62471fc69e61cdd1f6b4e91f41742b8b72d845058b51f16e519ecad48f0" Sep 08 16:53:34.106165 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.106138 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:34.110924 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.110909 2556 scope.go:117] "RemoveContainer" containerID="6819ec89697cea3764082ef80a75a5545ca770de640fac434dfa9141e19d0a86" Sep 08 16:53:34.118425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.118405 2556 scope.go:117] "RemoveContainer" containerID="e0bcdcd3c9d9c853a7833059d5157c9904130f6c721a80c506335559ffddbd94" Sep 08 16:53:34.118540 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.118521 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:34.182004 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.181976 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:34.182274 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182263 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="init-config-reloader" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182276 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="init-config-reloader" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182284 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182290 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182299 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="alertmanager" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182304 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="alertmanager" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182310 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="prom-label-proxy" Sep 08 16:53:34.182315 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182314 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="prom-label-proxy" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182336 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-web" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182342 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-web" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182349 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="config-reloader" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182354 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="config-reloader" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182365 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-metric" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182370 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-metric" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182415 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182423 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="prom-label-proxy" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182431 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="config-reloader" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182435 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-web" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182440 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="kube-rbac-proxy-metric" Sep 08 16:53:34.182533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.182446 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" containerName="alertmanager" Sep 08 16:53:34.185304 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.185289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.189550 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.189527 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 08 16:53:34.189639 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.189593 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 08 16:53:34.189639 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.189594 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 08 16:53:34.190037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.190002 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 08 16:53:34.190037 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.190007 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 08 16:53:34.190191 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.190137 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-524nd\"" Sep 08 16:53:34.190256 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.190230 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 08 16:53:34.190524 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.190501 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 08 16:53:34.192709 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.192686 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 08 16:53:34.194509 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.194492 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 08 16:53:34.212401 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.212354 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:34.258767 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258741 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.258767 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72v9t\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-kube-api-access-72v9t\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.258920 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258791 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.258920 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258809 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.258920 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259019 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-out\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259019 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258944 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259019 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.258987 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259019 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.259010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259156 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.259032 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259156 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.259053 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259156 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.259072 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-web-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.259156 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.259094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360285 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360433 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-out\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360433 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360315 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360433 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360361 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360433 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360391 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360621 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360621 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360558 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.360621 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.360590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-web-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.361176 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.361150 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.361347 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.361311 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.361451 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.361434 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.361509 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.361469 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72v9t\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-kube-api-access-72v9t\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.361754 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.361728 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.362101 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.362083 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.362233 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.362217 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.363633 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.363491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-out\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.363791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.363766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.363947 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.363924 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.364020 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.363984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.364580 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6573f9a7-a191-4457-89a8-3e93a99d27a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.364831 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.365245 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.366828 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-web-config\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.366956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6573f9a7-a191-4457-89a8-3e93a99d27a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.369086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.367124 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.371261 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.371233 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72v9t\" (UniqueName: \"kubernetes.io/projected/6573f9a7-a191-4457-89a8-3e93a99d27a5-kube-api-access-72v9t\") pod \"alertmanager-main-0\" (UID: \"6573f9a7-a191-4457-89a8-3e93a99d27a5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.494496 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.494421 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 08 16:53:34.789687 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:53:34.789634 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6573f9a7_a191_4457_89a8_3e93a99d27a5.slice/crio-7c6807673eb4a85317b3409bc9d83b02381a7de74a3f6f709c43a844e82d52f5 WatchSource:0}: Error finding container 7c6807673eb4a85317b3409bc9d83b02381a7de74a3f6f709c43a844e82d52f5: Status 404 returned error can't find the container with id 7c6807673eb4a85317b3409bc9d83b02381a7de74a3f6f709c43a844e82d52f5 Sep 08 16:53:34.792133 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:34.792114 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 08 16:53:35.080640 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:35.080565 2556 generic.go:356] "Generic (PLEG): container finished" podID="6573f9a7-a191-4457-89a8-3e93a99d27a5" containerID="9098b62cdb09fe6707cb9559388244562cbd167f570c993b951d61b58cf003af" exitCode=0 Sep 08 16:53:35.080640 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:35.080619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerDied","Data":"9098b62cdb09fe6707cb9559388244562cbd167f570c993b951d61b58cf003af"} Sep 08 16:53:35.081012 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:35.080667 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"7c6807673eb4a85317b3409bc9d83b02381a7de74a3f6f709c43a844e82d52f5"} Sep 08 16:53:35.342177 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:35.342151 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1231a9ff-3fd0-4b18-b006-815f0ebfe07a" path="/var/lib/kubelet/pods/1231a9ff-3fd0-4b18-b006-815f0ebfe07a/volumes" Sep 08 16:53:36.088430 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088393 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"a3a8dd11f28fd577fbd3881920aec3cc75718b715f1f1a36d7e2bf171c3406cd"} Sep 08 16:53:36.088800 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088474 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"7b6ee976b8d2ba722212ad0200470dd1b1e58012988ac4b96a351d14aa413916"} Sep 08 16:53:36.088800 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088514 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"cfdd37c668a84aa022b8ced2499a2da89cbf1442a8cca09715dff4b8f52e97ac"} Sep 08 16:53:36.088800 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088558 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"efad0ac4195763463ca6db80ff4e5396fef8c1c79df37f6b900c58ba49cbccfe"} Sep 08 16:53:36.088800 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088594 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"bee1c3b43ee72be47157a965918425802bd44711a5f0eab60350d5772c762f4b"} Sep 08 16:53:36.088800 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.088631 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6573f9a7-a191-4457-89a8-3e93a99d27a5","Type":"ContainerStarted","Data":"f44025a9f8c978f9f8bb974c2d2d22ce4355fbc4705f464eddd4bee59ecad506"} Sep 08 16:53:36.138047 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.137994 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.121929362 podStartE2EDuration="2.137977314s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.047952ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:36.128911529 +0000 UTC m=+147.273682091" watchObservedRunningTime="2026-09-08 16:53:36.137977314 +0000 UTC m=+147.282747875" Sep 08 16:53:36.178214 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178180 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:36.178611 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178561 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="prometheus" containerID="cri-o://43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" gracePeriod=600 Sep 08 16:53:36.178868 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178752 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy" containerID="cri-o://2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" gracePeriod=600 Sep 08 16:53:36.178868 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178822 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-web" containerID="cri-o://fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" gracePeriod=600 Sep 08 16:53:36.179023 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178907 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-thanos" containerID="cri-o://37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" gracePeriod=600 Sep 08 16:53:36.179023 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178921 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="config-reloader" containerID="cri-o://95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" gracePeriod=600 Sep 08 16:53:36.179023 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.178937 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="thanos-sidecar" containerID="cri-o://42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" gracePeriod=600 Sep 08 16:53:36.410305 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.410281 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:36.480362 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480306 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42bds\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480362 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480364 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480386 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480411 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480435 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480468 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480494 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480529 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480602 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480557 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480617 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480643 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480668 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480702 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480757 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480791 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480825 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480857 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.480916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480890 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.481309 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480921 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.481309 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.480963 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1\") pod \"14eac833-4f9d-4246-a697-d3f6f034fc69\" (UID: \"14eac833-4f9d-4246-a697-d3f6f034fc69\") " Sep 08 16:53:36.481513 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.481488 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.481695 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.481515 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.481790 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.481762 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.482113 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.482087 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:53:36.482258 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.482238 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.484268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.483436 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.484268 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.483519 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.484640 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.484603 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 16:53:36.487133 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.487079 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.487393 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.487365 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds" (OuterVolumeSpecName: "kube-api-access-42bds") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "kube-api-access-42bds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:53:36.487731 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.487697 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config" (OuterVolumeSpecName: "config") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.487833 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.487736 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out" (OuterVolumeSpecName: "config-out") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 16:53:36.488116 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.488016 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.488372 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.488259 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.488532 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.488431 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 16:53:36.488532 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.488499 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.488824 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.488797 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.489675 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.489639 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.489794 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.489660 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.499415 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.499391 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config" (OuterVolumeSpecName: "web-config") pod "14eac833-4f9d-4246-a697-d3f6f034fc69" (UID: "14eac833-4f9d-4246-a697-d3f6f034fc69"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 16:53:36.581795 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581758 2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-grpc-tls\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.581795 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581786 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-config-out\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.581795 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581799 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581811 2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-kube-rbac-proxy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581824 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-metrics-client-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581836 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-db\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581847 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581859 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581871 2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581885 2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581899 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581911 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42bds\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-kube-api-access-42bds\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581923 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581933 2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-config\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581944 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-web-config\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581956 2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-metrics-client-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581968 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14eac833-4f9d-4246-a697-d3f6f034fc69-tls-assets\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581981 2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14eac833-4f9d-4246-a697-d3f6f034fc69-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.581995 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:36.582062 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:36.582009 2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14eac833-4f9d-4246-a697-d3f6f034fc69-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 16:53:37.095917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095881 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" exitCode=0 Sep 08 16:53:37.095917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095910 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" exitCode=0 Sep 08 16:53:37.095917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095919 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" exitCode=0 Sep 08 16:53:37.095917 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095926 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" exitCode=0 Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095935 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" exitCode=0 Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095942 2556 generic.go:356] "Generic (PLEG): container finished" podID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" exitCode=0 Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095952 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.095976 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096047 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096077 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096110 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096131 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} Sep 08 16:53:37.096492 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.096204 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14eac833-4f9d-4246-a697-d3f6f034fc69","Type":"ContainerDied","Data":"d91af50ef63631dce208bf4c206ea442d3eae8ba17046fd23641ec10341c58f4"} Sep 08 16:53:37.104881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.104836 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.114164 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.114141 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.121017 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.121001 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.127405 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.127373 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.132957 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.132933 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:37.134757 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.134743 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.141640 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.141620 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.148226 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.148210 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.148522 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.148501 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.148585 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.148535 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.148654 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.148589 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.148845 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.148827 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.148886 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.148851 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.148886 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.148865 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.149096 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.149075 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.149155 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149105 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.149155 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149127 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.149373 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.149356 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.149425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149379 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.149425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149403 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.149680 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.149664 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.149751 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149686 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.149751 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.149705 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.150033 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.150013 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.150087 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150038 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.150087 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150057 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.150358 ip-10-0-132-198 kubenswrapper[2556]: E0908 16:53:37.150305 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.150358 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150361 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.150358 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150379 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.150691 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150667 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.150691 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.150692 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.151129 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151099 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.151129 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151125 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.151443 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151407 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.151443 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151433 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.151651 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151631 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.151702 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151651 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.151935 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151908 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.152030 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.151936 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.152207 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152189 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.152258 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152209 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.152485 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152467 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.152542 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152485 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.152750 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152727 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.152815 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152749 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.152993 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152976 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.153042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.152992 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.153209 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153189 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.153209 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153208 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.153496 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153480 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.153496 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153496 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.153613 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153585 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:37.153746 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153728 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.153791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153747 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.153981 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153963 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.154027 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.153982 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.154169 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154154 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.154222 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154168 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.154401 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154382 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.154440 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154402 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.154638 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154622 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.154683 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154639 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.154831 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154815 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.154872 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.154831 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.155042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155026 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.155090 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155044 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.155251 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155236 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.155312 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155251 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.155491 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155475 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.155538 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155491 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.155697 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155682 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.155740 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155697 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.155916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155898 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.155916 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.155915 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.156140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156123 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.156140 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156140 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.156377 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156363 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.156377 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156377 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.156595 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156576 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.156642 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156596 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.156793 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156771 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.156793 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156792 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.156984 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156967 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.156984 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.156982 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.157207 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157192 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.157207 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157206 2556 scope.go:117] "RemoveContainer" containerID="37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7" Sep 08 16:53:37.157434 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157418 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7"} err="failed to get container status \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": rpc error: code = NotFound desc = could not find container \"37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7\": container with ID starting with 37d4aaddb41ec04f8d2a0b1ff5fe1bc2598774742a353ff3182df86a8b4a48a7 not found: ID does not exist" Sep 08 16:53:37.157434 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157433 2556 scope.go:117] "RemoveContainer" containerID="2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b" Sep 08 16:53:37.157629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157608 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b"} err="failed to get container status \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": rpc error: code = NotFound desc = could not find container \"2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b\": container with ID starting with 2cd5b9fcce1f245851f2a85a00cbe8b9b723b0175fa8cf4efe117a904804117b not found: ID does not exist" Sep 08 16:53:37.157679 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157629 2556 scope.go:117] "RemoveContainer" containerID="fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40" Sep 08 16:53:37.157809 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157794 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40"} err="failed to get container status \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": rpc error: code = NotFound desc = could not find container \"fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40\": container with ID starting with fb25ad00f84c6c2c99b65e83dcd4a0e56385b0cb7ec0ad4dbd72c5ff57d5fa40 not found: ID does not exist" Sep 08 16:53:37.157809 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157808 2556 scope.go:117] "RemoveContainer" containerID="42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034" Sep 08 16:53:37.158008 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.157993 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034"} err="failed to get container status \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": rpc error: code = NotFound desc = could not find container \"42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034\": container with ID starting with 42dcc80ee7c00a8ea2e575f1fbd0e4c93b58e2d15dfcf112cc38256b4fe50034 not found: ID does not exist" Sep 08 16:53:37.158008 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158007 2556 scope.go:117] "RemoveContainer" containerID="95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157" Sep 08 16:53:37.158194 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158179 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157"} err="failed to get container status \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": rpc error: code = NotFound desc = could not find container \"95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157\": container with ID starting with 95e8e75bccee4c11d05151a813b8febdd693ee8bced088580e8c44e9052aa157 not found: ID does not exist" Sep 08 16:53:37.158277 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158193 2556 scope.go:117] "RemoveContainer" containerID="43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8" Sep 08 16:53:37.158411 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158394 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8"} err="failed to get container status \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": rpc error: code = NotFound desc = could not find container \"43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8\": container with ID starting with 43daa4d35a73e98b88af1e03b4d7a05d1ad3fee61d7fa0840811be9627a1f2e8 not found: ID does not exist" Sep 08 16:53:37.158450 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158412 2556 scope.go:117] "RemoveContainer" containerID="1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a" Sep 08 16:53:37.158607 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.158591 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a"} err="failed to get container status \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": rpc error: code = NotFound desc = could not find container \"1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a\": container with ID starting with 1af41707f9fc424d46c85234a841a737f9c859289a071097309b00d8f699068a not found: ID does not exist" Sep 08 16:53:37.195073 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195048 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:37.195386 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195373 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="init-config-reloader" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195388 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="init-config-reloader" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195395 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-thanos" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195402 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-thanos" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195413 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="config-reloader" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195418 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="config-reloader" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195426 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="thanos-sidecar" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195431 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="thanos-sidecar" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195436 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy" Sep 08 16:53:37.195438 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195441 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195458 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="prometheus" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195463 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="prometheus" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195472 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-web" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195478 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-web" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195522 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="thanos-sidecar" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195530 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="prometheus" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195537 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-web" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195543 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195551 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="config-reloader" Sep 08 16:53:37.195724 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.195555 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" containerName="kube-rbac-proxy-thanos" Sep 08 16:53:37.200872 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.200852 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.204025 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.204004 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 08 16:53:37.204225 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.204206 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7lnj6\"" Sep 08 16:53:37.205954 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.205907 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 08 16:53:37.207052 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.207035 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 08 16:53:37.207233 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.207210 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 08 16:53:37.207233 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.207231 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 08 16:53:37.207419 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.207289 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 08 16:53:37.208957 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.208935 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 08 16:53:37.209054 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.208936 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 08 16:53:37.209054 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.208937 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 08 16:53:37.209253 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.209233 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 08 16:53:37.210366 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.210352 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 08 16:53:37.210883 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.210863 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vmlmlj5d1ul0\"" Sep 08 16:53:37.210993 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.210865 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 08 16:53:37.212639 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.212623 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 08 16:53:37.213862 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.213848 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 08 16:53:37.227278 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.227256 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:37.288729 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.288729 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.288931 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.288931 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288835 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.288931 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.288931 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288950 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-config-out\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.288989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289064 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289099 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289143 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289160 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289174 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289187 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289200 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzknn\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-kube-api-access-zzknn\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289248 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289461 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289264 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289461 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289288 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.289461 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.289303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-web-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.341746 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.341712 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14eac833-4f9d-4246-a697-d3f6f034fc69" path="/var/lib/kubelet/pods/14eac833-4f9d-4246-a697-d3f6f034fc69/volumes" Sep 08 16:53:37.390278 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390278 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390281 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390299 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-web-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390354 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390370 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390385 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390438 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390452 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-config-out\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390517 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390497 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390530 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390574 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390597 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390674 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390702 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390726 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzknn\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-kube-api-access-zzknn\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.390934 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.390758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.391415 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.391120 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.392932 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.391951 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.392932 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.392592 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.393540 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.393572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.393657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.393941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.393961 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74cb9b9e-8724-420b-9508-d3c2f775afc8-config-out\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-web-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394188 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394352 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394216 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394821 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394608 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394821 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394726 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74cb9b9e-8724-420b-9508-d3c2f775afc8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.394929 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.394867 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.395262 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.395236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.396224 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.396189 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.396596 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.396577 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.396876 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.396860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.396919 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.396872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74cb9b9e-8724-420b-9508-d3c2f775afc8-config\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.406667 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.406642 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzknn\" (UniqueName: \"kubernetes.io/projected/74cb9b9e-8724-420b-9508-d3c2f775afc8-kube-api-access-zzknn\") pod \"prometheus-k8s-0\" (UID: \"74cb9b9e-8724-420b-9508-d3c2f775afc8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.510974 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.510885 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:53:37.801744 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:37.801714 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 08 16:53:38.102457 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:38.102373 2556 generic.go:356] "Generic (PLEG): container finished" podID="74cb9b9e-8724-420b-9508-d3c2f775afc8" containerID="e350a301832747a64fe07e54f6ec52f2a56126705c7f6619a134d80b015591ef" exitCode=0 Sep 08 16:53:38.102794 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:38.102467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerDied","Data":"e350a301832747a64fe07e54f6ec52f2a56126705c7f6619a134d80b015591ef"} Sep 08 16:53:38.102794 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:38.102535 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"749fe97ba8c99b776fcb0dc6a7717b5d41e8c5f0841d3bbf37574359a95b9dfd"} Sep 08 16:53:39.110833 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110795 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"28b24ce641abb6e139ba54e1fb3969c51abe3d6e54253693283e734db0af8ea5"} Sep 08 16:53:39.111205 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110863 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"abdba0bf51cb5102934ad728534f5345b338aab245cc5cb95346ed46f5e45f1e"} Sep 08 16:53:39.111205 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110903 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"637434bcb3775e47fb4c5659ee76271cf7137b939aa21f8dd65902a8c4e164b7"} Sep 08 16:53:39.111205 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110931 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"d02204295e76cfe01f2adfc4c03f19b7795a939e663e91e4d3cda315bb9165eb"} Sep 08 16:53:39.111205 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110957 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"0119aaf825f0593313456067be14932e5c7cbd4366d942118e9c1b9c0c55d256"} Sep 08 16:53:39.111205 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.110995 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74cb9b9e-8724-420b-9508-d3c2f775afc8","Type":"ContainerStarted","Data":"bf03dc922feb878826e5e13be9b24f957249826ae7ee50642d2dc75fc9114cca"} Sep 08 16:53:39.152116 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:39.151568 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.130307506 podStartE2EDuration="2.151546822s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.239316ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:53:39.148489772 +0000 UTC m=+150.293260332" watchObservedRunningTime="2026-09-08 16:53:39.151546822 +0000 UTC m=+150.296317384" Sep 08 16:53:42.511914 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:53:42.511875 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:54:37.511162 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:37.511124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:54:37.526288 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:37.526263 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:54:38.354687 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:38.354662 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 08 16:54:51.620124 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.620090 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8bwkg"] Sep 08 16:54:51.625115 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.625093 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.633884 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.633867 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j7cxh\"" Sep 08 16:54:51.633969 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.633867 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 16:54:51.634814 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.634799 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 16:54:51.666599 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.666573 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8bwkg"] Sep 08 16:54:51.752922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.752886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.753069 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.752942 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qf6c\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-kube-api-access-9qf6c\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.854236 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.854204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.854382 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.854264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qf6c\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-kube-api-access-9qf6c\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.916157 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.916093 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qf6c\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-kube-api-access-9qf6c\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.923852 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.923826 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc87fe86-8d9f-47ec-92db-89643dacb45b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8bwkg\" (UID: \"fc87fe86-8d9f-47ec-92db-89643dacb45b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:51.948165 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:51.948144 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:52.242695 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:52.242619 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8bwkg"] Sep 08 16:54:52.276072 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:54:52.276047 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc87fe86_8d9f_47ec_92db_89643dacb45b.slice/crio-0e8bc8ef41b6d83105f2557c3921633984dbb258cddb27c20f872cad8b27f4f5 WatchSource:0}: Error finding container 0e8bc8ef41b6d83105f2557c3921633984dbb258cddb27c20f872cad8b27f4f5: Status 404 returned error can't find the container with id 0e8bc8ef41b6d83105f2557c3921633984dbb258cddb27c20f872cad8b27f4f5 Sep 08 16:54:52.399017 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:52.398984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" event={"ID":"fc87fe86-8d9f-47ec-92db-89643dacb45b","Type":"ContainerStarted","Data":"0e8bc8ef41b6d83105f2557c3921633984dbb258cddb27c20f872cad8b27f4f5"} Sep 08 16:54:56.417533 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:56.417486 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" event={"ID":"fc87fe86-8d9f-47ec-92db-89643dacb45b","Type":"ContainerStarted","Data":"0577ca71f2b2a114b7c047803eee760fd5e5a70e8bf4c0c7d750e5aaf5a29318"} Sep 08 16:54:56.417994 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:56.417552 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:54:56.439647 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:54:56.439601 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" podStartSLOduration=1.916044112 podStartE2EDuration="5.439562292s" totalImagesPullingTime="3.52351818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:54:56.438347156 +0000 UTC m=+227.583117713" watchObservedRunningTime="2026-09-08 16:54:56.439562292 +0000 UTC m=+227.584332853" Sep 08 16:55:02.425307 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:02.425273 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8bwkg" Sep 08 16:55:04.267187 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.267152 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-d5j2w"] Sep 08 16:55:04.270523 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.270505 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.274092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.274066 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hwsw6\"" Sep 08 16:55:04.289238 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.289143 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-d5j2w"] Sep 08 16:55:04.364639 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.364600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-bound-sa-token\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.364840 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.364679 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tf8\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-kube-api-access-j6tf8\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.465625 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.465594 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tf8\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-kube-api-access-j6tf8\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.465798 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.465686 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-bound-sa-token\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.477437 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.477397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tf8\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-kube-api-access-j6tf8\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.477617 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.477600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78-bound-sa-token\") pod \"cert-manager-55b4d954d5-d5j2w\" (UID: \"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78\") " pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.580651 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.580563 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-d5j2w" Sep 08 16:55:04.858786 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:04.858762 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-d5j2w"] Sep 08 16:55:04.861066 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:55:04.861038 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f5ff1b_9fb7_48f3_81b9_9e3f7d30ee78.slice/crio-6d5dbab94335fd0b657cedaf87bff4e1ae5fc1d827e51d06c1a1344b6486a864 WatchSource:0}: Error finding container 6d5dbab94335fd0b657cedaf87bff4e1ae5fc1d827e51d06c1a1344b6486a864: Status 404 returned error can't find the container with id 6d5dbab94335fd0b657cedaf87bff4e1ae5fc1d827e51d06c1a1344b6486a864 Sep 08 16:55:05.456908 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:05.456874 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-d5j2w" event={"ID":"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78","Type":"ContainerStarted","Data":"2857b9c7787c9a5726d9ee1ee081ead674cbd0d06723589abf50685b1d7d7c4c"} Sep 08 16:55:05.457273 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:05.456920 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-d5j2w" event={"ID":"49f5ff1b-9fb7-48f3-81b9-9e3f7d30ee78","Type":"ContainerStarted","Data":"6d5dbab94335fd0b657cedaf87bff4e1ae5fc1d827e51d06c1a1344b6486a864"} Sep 08 16:55:05.485946 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:05.485897 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-d5j2w" podStartSLOduration=1.4858812860000001 podStartE2EDuration="1.485881286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:05.484969634 +0000 UTC m=+236.629740196" watchObservedRunningTime="2026-09-08 16:55:05.485881286 +0000 UTC m=+236.630651847" Sep 08 16:55:24.278779 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.278747 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk"] Sep 08 16:55:24.285215 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.285193 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.326827 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.326797 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 16:55:24.326987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.326804 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 16:55:24.326987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.326866 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 16:55:24.326987 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.326933 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 16:55:24.327174 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.327161 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l98dr\"" Sep 08 16:55:24.373065 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.373037 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk"] Sep 08 16:55:24.434977 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.434939 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.435163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.434987 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btndp\" (UniqueName: \"kubernetes.io/projected/74f6407b-e10f-4bfc-8217-79cdf3706719-kube-api-access-btndp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.435163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.435117 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.435163 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.435148 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.435313 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.435207 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.535871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.535786 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btndp\" (UniqueName: \"kubernetes.io/projected/74f6407b-e10f-4bfc-8217-79cdf3706719-kube-api-access-btndp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.535871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.535845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.536092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.535959 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.536092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.536000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.536092 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.536049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.536389 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.536362 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.538039 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.538013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74f6407b-e10f-4bfc-8217-79cdf3706719-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.538249 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.538232 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.538362 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.538344 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74f6407b-e10f-4bfc-8217-79cdf3706719-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.554923 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.554902 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btndp\" (UniqueName: \"kubernetes.io/projected/74f6407b-e10f-4bfc-8217-79cdf3706719-kube-api-access-btndp\") pod \"opendatahub-operator-controller-manager-84f58c54b-wmfwk\" (UID: \"74f6407b-e10f-4bfc-8217-79cdf3706719\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.595641 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.595616 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:24.928247 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:55:24.928213 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f6407b_e10f_4bfc_8217_79cdf3706719.slice/crio-07de92178849ac59b9666c75823680fcb23fcf03947ac50fd446e84fd75795fd WatchSource:0}: Error finding container 07de92178849ac59b9666c75823680fcb23fcf03947ac50fd446e84fd75795fd: Status 404 returned error can't find the container with id 07de92178849ac59b9666c75823680fcb23fcf03947ac50fd446e84fd75795fd Sep 08 16:55:24.929279 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:24.929248 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk"] Sep 08 16:55:25.538979 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:25.538943 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" event={"ID":"74f6407b-e10f-4bfc-8217-79cdf3706719","Type":"ContainerStarted","Data":"07de92178849ac59b9666c75823680fcb23fcf03947ac50fd446e84fd75795fd"} Sep 08 16:55:28.555904 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:28.555873 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" event={"ID":"74f6407b-e10f-4bfc-8217-79cdf3706719","Type":"ContainerStarted","Data":"00f94505aa66e7deb0973da3c237508f11b3f7348a2518ab697eec9ba2561cf9"} Sep 08 16:55:29.561390 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:29.561297 2556 generic.go:356] "Generic (PLEG): container finished" podID="74f6407b-e10f-4bfc-8217-79cdf3706719" containerID="00f94505aa66e7deb0973da3c237508f11b3f7348a2518ab697eec9ba2561cf9" exitCode=0 Sep 08 16:55:29.561739 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:29.561386 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" event={"ID":"74f6407b-e10f-4bfc-8217-79cdf3706719","Type":"ContainerDied","Data":"00f94505aa66e7deb0973da3c237508f11b3f7348a2518ab697eec9ba2561cf9"} Sep 08 16:55:30.568136 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:30.568100 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" event={"ID":"74f6407b-e10f-4bfc-8217-79cdf3706719","Type":"ContainerStarted","Data":"c11c41a575dd726ad4f070e5567d11db9b3b7c1f770b6233b1d9736615ef876a"} Sep 08 16:55:30.568538 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:30.568294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:30.611173 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:30.611094 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" podStartSLOduration=1.687367918 podStartE2EDuration="6.611079643s" totalImagesPullingTime="3.853860224s" totalInitContainerRuntime="1.069851501s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:30.610574364 +0000 UTC m=+261.755344924" watchObservedRunningTime="2026-09-08 16:55:30.611079643 +0000 UTC m=+261.755850203" Sep 08 16:55:41.575607 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:41.575527 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-wmfwk" Sep 08 16:55:54.713473 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.713442 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7"] Sep 08 16:55:54.720784 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.720760 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.725084 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.725063 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 16:55:54.725948 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.725927 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 16:55:54.726076 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.725985 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tm958\"" Sep 08 16:55:54.726847 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.726831 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 16:55:54.742537 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.742513 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7"] Sep 08 16:55:54.789674 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.789674 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.789881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789694 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfskm\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-kube-api-access-sfskm\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.789881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789823 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.789881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789864 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.789881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789881 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.790067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789906 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.790067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.789955 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.790067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.790022 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891471 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891655 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891655 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891655 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfskm\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-kube-api-access-sfskm\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891825 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891721 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891825 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891825 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.891825 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892017 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892017 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.891913 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892122 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.892076 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892215 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.892196 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892267 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.892199 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.892497 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.892479 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.893914 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.893893 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.894006 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.893923 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.904519 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.904500 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfskm\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-kube-api-access-sfskm\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:54.904761 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:54.904743 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18f8aa4-b156-48a3-ac6b-87405aaad7c6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7\" (UID: \"c18f8aa4-b156-48a3-ac6b-87405aaad7c6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:55.030952 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:55.030879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:55.385221 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:55:55.385196 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18f8aa4_b156_48a3_ac6b_87405aaad7c6.slice/crio-d0a407607efa277e6f6017199bcc1586b95af6e2c8eb9dc53b7a10b7eb76ff05 WatchSource:0}: Error finding container d0a407607efa277e6f6017199bcc1586b95af6e2c8eb9dc53b7a10b7eb76ff05: Status 404 returned error can't find the container with id d0a407607efa277e6f6017199bcc1586b95af6e2c8eb9dc53b7a10b7eb76ff05 Sep 08 16:55:55.391960 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:55.391937 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7"] Sep 08 16:55:55.683111 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:55.683033 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" event={"ID":"c18f8aa4-b156-48a3-ac6b-87405aaad7c6","Type":"ContainerStarted","Data":"d0a407607efa277e6f6017199bcc1586b95af6e2c8eb9dc53b7a10b7eb76ff05"} Sep 08 16:55:58.294354 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:58.294290 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:55:58.294651 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:58.294386 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:55:58.294651 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:58.294419 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:55:58.700925 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:58.700891 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" event={"ID":"c18f8aa4-b156-48a3-ac6b-87405aaad7c6","Type":"ContainerStarted","Data":"b62111056bcaf8027e411f120028d58ccf464a7c966bd243644b9dec8e38dde2"} Sep 08 16:55:59.031286 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:59.031202 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:59.036095 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:59.036069 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:59.076382 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:59.076334 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" podStartSLOduration=2.169283269 podStartE2EDuration="5.076305313s" totalImagesPullingTime="2.907022044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:58.734167089 +0000 UTC m=+289.878937661" watchObservedRunningTime="2026-09-08 16:55:59.076305313 +0000 UTC m=+290.221075873" Sep 08 16:55:59.705935 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:59.705902 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:55:59.706978 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:55:59.706955 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7" Sep 08 16:56:03.459444 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.459414 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh"] Sep 08 16:56:03.463067 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.463042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.467686 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.467660 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 16:56:03.468799 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.468776 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 16:56:03.468919 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.468869 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 16:56:03.468984 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.468933 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 16:56:03.469109 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.469089 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xfgpt\"" Sep 08 16:56:03.473288 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.473270 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 16:56:03.500629 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.500603 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh"] Sep 08 16:56:03.573899 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.573868 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.574044 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.573903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtr7\" (UniqueName: \"kubernetes.io/projected/f850b741-e351-4a1d-8adf-f802652192ce-kube-api-access-tmtr7\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.574044 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.573983 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.574044 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.574013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f850b741-e351-4a1d-8adf-f802652192ce-manager-config\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.674817 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.674777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.674817 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.674818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtr7\" (UniqueName: \"kubernetes.io/projected/f850b741-e351-4a1d-8adf-f802652192ce-kube-api-access-tmtr7\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.675080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.674860 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.675080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.674885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f850b741-e351-4a1d-8adf-f802652192ce-manager-config\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.675589 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.675566 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f850b741-e351-4a1d-8adf-f802652192ce-manager-config\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.677273 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.677252 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.677369 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.677292 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f850b741-e351-4a1d-8adf-f802652192ce-cert\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.687577 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.687556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtr7\" (UniqueName: \"kubernetes.io/projected/f850b741-e351-4a1d-8adf-f802652192ce-kube-api-access-tmtr7\") pod \"lws-controller-manager-74fc5c54f8-sx7nh\" (UID: \"f850b741-e351-4a1d-8adf-f802652192ce\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:03.775520 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:03.775448 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:04.071950 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.071919 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh"] Sep 08 16:56:04.073074 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:56:04.073044 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf850b741_e351_4a1d_8adf_f802652192ce.slice/crio-ecc507bf0d0bf1ba10ab12b8848699039ad4738726dd367b1d41604ef4d4af8d WatchSource:0}: Error finding container ecc507bf0d0bf1ba10ab12b8848699039ad4738726dd367b1d41604ef4d4af8d: Status 404 returned error can't find the container with id ecc507bf0d0bf1ba10ab12b8848699039ad4738726dd367b1d41604ef4d4af8d Sep 08 16:56:04.467052 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.467014 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm"] Sep 08 16:56:04.472121 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.472094 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.475173 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.475151 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 16:56:04.475310 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.475155 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 16:56:04.478642 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.478622 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 16:56:04.478871 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.478857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dsdcl\"" Sep 08 16:56:04.499378 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.498754 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm"] Sep 08 16:56:04.583184 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583150 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583214 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7r8\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-kube-api-access-fk7r8\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.583398 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.583381 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684423 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684387 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684423 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684464 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684479 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7r8\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-kube-api-access-fk7r8\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.684660 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.684520 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.685357 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.685310 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.686881 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.686858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.686980 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.686896 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.687080 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.687062 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.687191 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.687175 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.694968 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.694946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.699749 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.699729 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7r8\" (UniqueName: \"kubernetes.io/projected/c1dda4eb-600b-4b7e-ba4c-546ae562fff5-kube-api-access-fk7r8\") pod \"istiod-openshift-gateway-5677ccb49b-njtwm\" (UID: \"c1dda4eb-600b-4b7e-ba4c-546ae562fff5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:04.727953 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.727886 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" event={"ID":"f850b741-e351-4a1d-8adf-f802652192ce","Type":"ContainerStarted","Data":"ecc507bf0d0bf1ba10ab12b8848699039ad4738726dd367b1d41604ef4d4af8d"} Sep 08 16:56:04.782963 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:04.782931 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:05.172457 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:05.172376 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm"] Sep 08 16:56:05.176259 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:56:05.176221 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dda4eb_600b_4b7e_ba4c_546ae562fff5.slice/crio-e0e0c36514ca49da1efe07345285ed4beea6bbaaf5a237efff13fb01eaa4f25b WatchSource:0}: Error finding container e0e0c36514ca49da1efe07345285ed4beea6bbaaf5a237efff13fb01eaa4f25b: Status 404 returned error can't find the container with id e0e0c36514ca49da1efe07345285ed4beea6bbaaf5a237efff13fb01eaa4f25b Sep 08 16:56:05.734958 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:05.734912 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" event={"ID":"c1dda4eb-600b-4b7e-ba4c-546ae562fff5","Type":"ContainerStarted","Data":"e0e0c36514ca49da1efe07345285ed4beea6bbaaf5a237efff13fb01eaa4f25b"} Sep 08 16:56:06.742692 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:06.742651 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" event={"ID":"f850b741-e351-4a1d-8adf-f802652192ce","Type":"ContainerStarted","Data":"1a08ff6c6ab07299ed7f6f1abe7be39867ab495d50dcad156e2551ff25146fc2"} Sep 08 16:56:06.743206 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:06.742892 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:56:06.783199 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:06.782822 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" podStartSLOduration=1.349316546 podStartE2EDuration="3.782800914s" totalImagesPullingTime="2.433484368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:06.781182012 +0000 UTC m=+297.925952573" watchObservedRunningTime="2026-09-08 16:56:06.782800914 +0000 UTC m=+297.927571476" Sep 08 16:56:08.246503 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:08.246463 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:56:08.246741 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:08.246531 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:56:08.754995 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:08.754957 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" event={"ID":"c1dda4eb-600b-4b7e-ba4c-546ae562fff5","Type":"ContainerStarted","Data":"5c2903666230d5a96fbc9e208bf0291f62d8367a818169855b28ed89a31f62cf"} Sep 08 16:56:08.755172 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:08.755051 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:08.811977 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:08.811918 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" podStartSLOduration=1.7446634250000002 podStartE2EDuration="4.811899027s" totalImagesPullingTime="3.067235602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:08.809616775 +0000 UTC m=+299.954387337" watchObservedRunningTime="2026-09-08 16:56:08.811899027 +0000 UTC m=+299.956669589" Sep 08 16:56:09.247193 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:09.247150 2556 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 16:56:09.760786 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:09.760739 2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-njtwm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 16:56:09.762978 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:09.760808 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" podUID="c1dda4eb-600b-4b7e-ba4c-546ae562fff5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 16:56:12.760715 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:12.760679 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-njtwm" Sep 08 16:56:17.750702 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:56:17.750670 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-sx7nh" Sep 08 16:58:16.794437 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.794398 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929"] Sep 08 16:58:16.797875 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.797856 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.802231 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.802209 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bcq5x\"" Sep 08 16:58:16.815364 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.815340 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929"] Sep 08 16:58:16.878182 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878150 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878259 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdtt\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-kube-api-access-xbdtt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878367 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878343 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878538 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878387 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878538 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878417 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.878538 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.878439 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.979885 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.979847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980049 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.979894 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980049 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980022 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980122 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdtt\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-kube-api-access-xbdtt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980122 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980100 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980199 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980199 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980156 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980297 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980242 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980432 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980301 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980491 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980633 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980599 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980710 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.980770 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.980716 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.982425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.982407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.982583 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.982565 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.989687 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.989664 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:16.989805 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:16.989786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdtt\" (UniqueName: \"kubernetes.io/projected/deb2cf38-f822-4c65-a4f9-94eaac8469b6-kube-api-access-xbdtt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h9929\" (UID: \"deb2cf38-f822-4c65-a4f9-94eaac8469b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:17.108936 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:17.108850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:17.401596 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:17.401569 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929"] Sep 08 16:58:17.404576 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:58:17.404550 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb2cf38_f822_4c65_a4f9_94eaac8469b6.slice/crio-440a2fd6998472d0fc8e31f94ccd23b9d6aea45e31d806cabda45f0922256ee8 WatchSource:0}: Error finding container 440a2fd6998472d0fc8e31f94ccd23b9d6aea45e31d806cabda45f0922256ee8: Status 404 returned error can't find the container with id 440a2fd6998472d0fc8e31f94ccd23b9d6aea45e31d806cabda45f0922256ee8 Sep 08 16:58:17.407015 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:17.406982 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:58:17.407090 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:17.407049 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:58:17.407090 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:17.407079 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:58:18.332425 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:18.332387 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" event={"ID":"deb2cf38-f822-4c65-a4f9-94eaac8469b6","Type":"ContainerStarted","Data":"850c20fe4d3971e0d01721143eec5b318aa1e317e37b20021106981a0a67ea25"} Sep 08 16:58:18.332850 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:18.332466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" event={"ID":"deb2cf38-f822-4c65-a4f9-94eaac8469b6","Type":"ContainerStarted","Data":"440a2fd6998472d0fc8e31f94ccd23b9d6aea45e31d806cabda45f0922256ee8"} Sep 08 16:58:18.373363 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:18.373291 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podStartSLOduration=2.373276043 podStartE2EDuration="2.373276043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:18.373232721 +0000 UTC m=+429.518003282" watchObservedRunningTime="2026-09-08 16:58:18.373276043 +0000 UTC m=+429.518046604" Sep 08 16:58:19.109167 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.109130 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:19.113704 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.113679 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:19.341241 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.341211 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:19.341622 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.341278 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:58:19.901414 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.901376 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 16:58:19.904841 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.904825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:19.911251 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.911231 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 16:58:19.911929 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.911870 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 16:58:19.912308 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.912288 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 16:58:19.913251 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.913233 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9kmwg\"" Sep 08 16:58:19.925573 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.925551 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 16:58:19.970421 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:19.970392 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 16:58:20.007922 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.007892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzx7\" (UniqueName: \"kubernetes.io/projected/51772def-5c84-49be-ba3c-8498a8fea26d-kube-api-access-wtzx7\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.008035 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.007929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/51772def-5c84-49be-ba3c-8498a8fea26d-config-file\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.108670 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.108641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzx7\" (UniqueName: \"kubernetes.io/projected/51772def-5c84-49be-ba3c-8498a8fea26d-kube-api-access-wtzx7\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.108791 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.108675 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/51772def-5c84-49be-ba3c-8498a8fea26d-config-file\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.109222 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.109203 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/51772def-5c84-49be-ba3c-8498a8fea26d-config-file\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.121971 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.121941 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzx7\" (UniqueName: \"kubernetes.io/projected/51772def-5c84-49be-ba3c-8498a8fea26d-kube-api-access-wtzx7\") pod \"limitador-limitador-84994f7ffc-4wmfx\" (UID: \"51772def-5c84-49be-ba3c-8498a8fea26d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.215732 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.215667 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:20.496168 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.495930 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 16:58:20.498227 ip-10-0-132-198 kubenswrapper[2556]: W0908 16:58:20.498198 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51772def_5c84_49be_ba3c_8498a8fea26d.slice/crio-7506d1c626424ba1cbf3af725c3c9e46667b49135601d81f1589ad81a9163429 WatchSource:0}: Error finding container 7506d1c626424ba1cbf3af725c3c9e46667b49135601d81f1589ad81a9163429: Status 404 returned error can't find the container with id 7506d1c626424ba1cbf3af725c3c9e46667b49135601d81f1589ad81a9163429 Sep 08 16:58:20.500588 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:20.500570 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 16:58:21.349086 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:21.349055 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" event={"ID":"51772def-5c84-49be-ba3c-8498a8fea26d","Type":"ContainerStarted","Data":"7506d1c626424ba1cbf3af725c3c9e46667b49135601d81f1589ad81a9163429"} Sep 08 16:58:24.366683 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:24.366644 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" event={"ID":"51772def-5c84-49be-ba3c-8498a8fea26d","Type":"ContainerStarted","Data":"2f3acf1ed0f468a5ff75f19ea4777451d07ba80500da83cdc2ad58879780436e"} Sep 08 16:58:24.367058 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:24.366786 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:58:24.388042 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:24.387989 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" podStartSLOduration=2.321381566 podStartE2EDuration="5.387973873s" totalImagesPullingTime="3.066592307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:58:24.386174593 +0000 UTC m=+435.530945155" watchObservedRunningTime="2026-09-08 16:58:24.387973873 +0000 UTC m=+435.532744500" Sep 08 16:58:35.372625 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:58:35.372548 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-4wmfx" Sep 08 16:59:36.258898 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:36.258858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 16:59:45.751181 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.751141 2556 generic.go:356] "Generic (PLEG): container finished" podID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerID="850c20fe4d3971e0d01721143eec5b318aa1e317e37b20021106981a0a67ea25" exitCode=0 Sep 08 16:59:45.751640 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.751220 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" event={"ID":"deb2cf38-f822-4c65-a4f9-94eaac8469b6","Type":"ContainerDied","Data":"850c20fe4d3971e0d01721143eec5b318aa1e317e37b20021106981a0a67ea25"} Sep 08 16:59:45.751698 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.751649 2556 scope.go:117] "RemoveContainer" containerID="850c20fe4d3971e0d01721143eec5b318aa1e317e37b20021106981a0a67ea25" Sep 08 16:59:45.752720 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.752695 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:59:45.752802 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.752767 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:59:45.752839 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:45.752803 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 16:59:46.760988 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:46.760951 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" event={"ID":"deb2cf38-f822-4c65-a4f9-94eaac8469b6","Type":"ContainerStarted","Data":"7e7939b5cfe869a4b5d42466eec66c2b088c397cdb46e431025046348a03e1f8"} Sep 08 16:59:47.109351 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:47.109249 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:59:47.110342 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:47.110293 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h9929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 08 16:59:47.110457 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:47.110372 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podUID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 08 16:59:48.109711 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:48.109675 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h9929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 08 16:59:48.110089 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:48.109730 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podUID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 08 16:59:49.110229 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:49.110193 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h9929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 08 16:59:49.110626 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:49.110245 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podUID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 08 16:59:49.341941 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:49.341912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:59:50.109658 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:50.109619 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h9929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 08 16:59:50.109861 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:50.109697 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podUID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 08 16:59:51.109963 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:51.109928 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h9929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" start-of-body= Sep 08 16:59:51.110336 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:51.109988 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" podUID="deb2cf38-f822-4c65-a4f9-94eaac8469b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.26:15021/healthz/ready\": dial tcp 10.134.0.26:15021: connect: connection refused" Sep 08 16:59:52.113446 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:52.113405 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 16:59:52.114484 ip-10-0-132-198 kubenswrapper[2556]: I0908 16:59:52.114464 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h9929" Sep 08 17:00:03.110708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:03.110618 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:00:15.026304 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:15.026271 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:00:28.612389 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:28.612351 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:00:38.424528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.424496 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl"] Sep 08 17:00:38.426907 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.426883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.432179 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.432138 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 08 17:00:38.432306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.432196 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 17:00:38.432306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.432222 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9fdqs\"" Sep 08 17:00:38.432306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.432243 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 17:00:38.442223 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.442201 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl"] Sep 08 17:00:38.541583 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541554 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.541583 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541585 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.541829 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541608 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.541829 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541628 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.541829 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541787 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmksb\" (UniqueName: \"kubernetes.io/projected/79074beb-51f3-4b8d-8c7d-675a08ef8a34-kube-api-access-jmksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.541962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.541836 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642734 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642697 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmksb\" (UniqueName: \"kubernetes.io/projected/79074beb-51f3-4b8d-8c7d-675a08ef8a34-kube-api-access-jmksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642771 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642786 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642807 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.642903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.642830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.643265 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.643243 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.643350 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.643263 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.643350 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.643294 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.645143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.645121 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79074beb-51f3-4b8d-8c7d-675a08ef8a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.645426 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.645408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79074beb-51f3-4b8d-8c7d-675a08ef8a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.651894 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.651870 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmksb\" (UniqueName: \"kubernetes.io/projected/79074beb-51f3-4b8d-8c7d-675a08ef8a34-kube-api-access-jmksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wkbhl\" (UID: \"79074beb-51f3-4b8d-8c7d-675a08ef8a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:38.738240 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:38.738144 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:39.011760 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:39.011689 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl"] Sep 08 17:00:39.015209 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:00:39.015183 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79074beb_51f3_4b8d_8c7d_675a08ef8a34.slice/crio-dbff908555dd05ef1db0b7ce4894043dfb369d685b96998225e59813a0a11053 WatchSource:0}: Error finding container dbff908555dd05ef1db0b7ce4894043dfb369d685b96998225e59813a0a11053: Status 404 returned error can't find the container with id dbff908555dd05ef1db0b7ce4894043dfb369d685b96998225e59813a0a11053 Sep 08 17:00:39.614068 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:39.614032 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:00:40.007195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:40.007168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" event={"ID":"79074beb-51f3-4b8d-8c7d-675a08ef8a34","Type":"ContainerStarted","Data":"dbff908555dd05ef1db0b7ce4894043dfb369d685b96998225e59813a0a11053"} Sep 08 17:00:41.013606 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:41.013573 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" event={"ID":"79074beb-51f3-4b8d-8c7d-675a08ef8a34","Type":"ContainerStarted","Data":"755711048e6191b0d1fdf6d71c39311259979c748bd5220628c68362a5d6c4c6"} Sep 08 17:00:41.036867 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:41.036820 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" podStartSLOduration=2.053783427 podStartE2EDuration="3.036802549s" totalImagesPullingTime="983.019122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:41.036487418 +0000 UTC m=+572.181257983" watchObservedRunningTime="2026-09-08 17:00:41.036802549 +0000 UTC m=+572.181573110" Sep 08 17:00:48.738913 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:48.738872 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:48.739414 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:48.738945 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:48.751491 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:48.751468 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:00:49.063006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:00:49.062930 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wkbhl" Sep 08 17:01:02.708487 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:01:02.708450 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:01:12.112143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:01:12.112103 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:01:36.126192 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:01:36.126116 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:02:59.136423 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.136390 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:02:59.139791 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.139765 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.144742 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.144722 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2746ad-openshift-default-dockercfg-m4gh8\"" Sep 08 17:02:59.206984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.206955 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:02:59.247036 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.246998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42rs\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247036 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247221 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247059 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247221 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247163 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247221 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247210 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247341 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247236 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247341 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247270 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247410 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.247519 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.247492 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348059 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348059 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42rs\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348082 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348130 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348174 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348210 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348264 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348296 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348483 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348511 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348653 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.348731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348730 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.349003 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.348984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.350409 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.350390 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.350534 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.350518 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.368113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.368089 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.409112 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.409047 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42rs\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs\") pod \"e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.450944 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.450919 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:02:59.758567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.758475 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:02:59.762945 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:02:59.762913 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb70d0c_dfde_4ee4_a80d_8f717beffc7a.slice/crio-7e5e82d4367ad5361efdce1a92fb3e7c8a19b8a9cc58451c08c6291958da1bdf WatchSource:0}: Error finding container 7e5e82d4367ad5361efdce1a92fb3e7c8a19b8a9cc58451c08c6291958da1bdf: Status 404 returned error can't find the container with id 7e5e82d4367ad5361efdce1a92fb3e7c8a19b8a9cc58451c08c6291958da1bdf Sep 08 17:02:59.765461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.765426 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:02:59.765550 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.765502 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:02:59.765550 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:02:59.765532 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:00.663117 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:00.663082 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerStarted","Data":"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd"} Sep 08 17:03:00.663506 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:00.663161 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerStarted","Data":"7e5e82d4367ad5361efdce1a92fb3e7c8a19b8a9cc58451c08c6291958da1bdf"} Sep 08 17:03:00.747391 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:00.747338 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podStartSLOduration=1.747308039 podStartE2EDuration="1.747308039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:00.746744221 +0000 UTC m=+711.891514784" watchObservedRunningTime="2026-09-08 17:03:00.747308039 +0000 UTC m=+711.892078599" Sep 08 17:03:01.451772 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:01.451742 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:01.456308 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:01.456283 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:01.667662 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:01.667624 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:01.668638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:01.668620 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:03.071864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:03.071830 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:03:08.735940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.735909 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:03:08.740005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.739984 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.743387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.743365 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a57edc2e-openshift-default-dockercfg-7pb6q\"" Sep 08 17:03:08.775949 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.775922 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:03:08.829251 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829214 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rfk\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829251 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829255 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829281 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829410 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829464 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829488 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.829701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.829557 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930554 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930613 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930683 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rfk\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930721 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930807 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.930927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930908 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.931122 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.930943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.931476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.931395 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.931603 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.931538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.932194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.931911 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.932194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.931960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.932623 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.932601 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.934262 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.934237 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.934385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.934308 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.951684 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.951648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:08.951833 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:08.951820 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rfk\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk\") pod \"e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:09.053184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.053107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:09.383609 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.383531 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:03:09.385560 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:09.385535 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d479400_7cd9_477d_af35_0380bdc7fd76.slice/crio-0b014f9672f2b362ce90db28078a683edeb5a445fbe4c7701e33053f9ff92ac0 WatchSource:0}: Error finding container 0b014f9672f2b362ce90db28078a683edeb5a445fbe4c7701e33053f9ff92ac0: Status 404 returned error can't find the container with id 0b014f9672f2b362ce90db28078a683edeb5a445fbe4c7701e33053f9ff92ac0 Sep 08 17:03:09.388550 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.388511 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:09.388736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.388707 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:09.388887 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.388768 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:09.708316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.708233 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerStarted","Data":"a5f03abff40289b6d459af466fe7cf22357b9f24f13dc8f2ae5179c0d9007094"} Sep 08 17:03:09.708316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.708283 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerStarted","Data":"0b014f9672f2b362ce90db28078a683edeb5a445fbe4c7701e33053f9ff92ac0"} Sep 08 17:03:09.783374 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:09.783306 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podStartSLOduration=1.783288677 podStartE2EDuration="1.783288677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:09.780187886 +0000 UTC m=+720.924958446" watchObservedRunningTime="2026-09-08 17:03:09.783288677 +0000 UTC m=+720.928059305" Sep 08 17:03:10.054362 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:10.054242 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:10.064570 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:10.064535 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:10.714820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:10.714774 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:10.715852 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:10.715827 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:13.789895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:13.789863 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:03:14.558343 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.557911 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:03:14.571730 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.571706 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.575473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.575423 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-66e7a8-openshift-default-dockercfg-c98w8\"" Sep 08 17:03:14.576707 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.576655 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:03:14.588116 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588131 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588158 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2v5\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588188 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588209 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588287 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.588390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.588334 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.689921 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2v5\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690010 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690079 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690133 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690906 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690364 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690906 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690584 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690906 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690642 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.690906 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.690897 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.691121 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.691045 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.691217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.691195 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.693431 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.693409 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.693567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.693432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.707173 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.707135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2v5\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.707447 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.707429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token\") pod \"e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:14.886804 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:14.886770 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:15.196307 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.196098 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:03:15.198883 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:15.198859 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace42ef_e47e_46de_9e0c_7c75c486ccd4.slice/crio-4491ec57da0207bc38e0cfdfa5419c36c8727c924c472a74a392f647a882298c WatchSource:0}: Error finding container 4491ec57da0207bc38e0cfdfa5419c36c8727c924c472a74a392f647a882298c: Status 404 returned error can't find the container with id 4491ec57da0207bc38e0cfdfa5419c36c8727c924c472a74a392f647a882298c Sep 08 17:03:15.201092 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.201063 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:15.201158 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.201123 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:15.201158 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.201151 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:15.741253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.741213 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerStarted","Data":"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777"} Sep 08 17:03:15.741469 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.741277 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerStarted","Data":"4491ec57da0207bc38e0cfdfa5419c36c8727c924c472a74a392f647a882298c"} Sep 08 17:03:15.887502 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:15.887466 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:16.892902 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:16.892872 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:16.926361 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:16.926294 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podStartSLOduration=2.926278497 podStartE2EDuration="2.926278497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:15.779633045 +0000 UTC m=+726.924403606" watchObservedRunningTime="2026-09-08 17:03:16.926278497 +0000 UTC m=+728.071049059" Sep 08 17:03:17.710498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:17.710455 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 17:03:17.710681 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:17.710525 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 17:03:17.752049 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:17.752021 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:17.752937 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:17.752912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:18.757886 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.757781 2556 generic.go:356] "Generic (PLEG): container finished" podID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerID="570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd" exitCode=0 Sep 08 17:03:18.757886 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.757851 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerDied","Data":"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd"} Sep 08 17:03:18.758484 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.758457 2556 scope.go:117] "RemoveContainer" containerID="570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd" Sep 08 17:03:18.759515 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.759485 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:18.759612 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.759578 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:18.759661 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:18.759619 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:19.779995 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:19.779960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerStarted","Data":"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97"} Sep 08 17:03:20.451485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:20.451451 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:20.451792 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:20.451497 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 08 17:03:20.451982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:20.451950 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 08 17:03:21.451766 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:21.451724 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 08 17:03:21.452244 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:21.451809 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 08 17:03:22.452150 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:22.452102 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 08 17:03:22.452669 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:22.452170 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 08 17:03:23.452103 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:23.452065 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 08 17:03:23.452413 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:23.452380 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 08 17:03:24.451698 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:24.451663 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 08 17:03:24.452011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:24.451969 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 08 17:03:25.462743 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:25.462697 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:25.463556 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:25.463528 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:25.468643 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:25.468612 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:03:27.588649 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:27.588612 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:03:28.790970 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.790935 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:03:28.811946 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.811382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.823206 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.823179 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5a7cab-openshift-default-dockercfg-ktg5c\"" Sep 08 17:03:28.875881 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.875838 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.957896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.957954 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958021 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958049 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958112 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhsg5\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958165 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958216 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958280 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:28.958481 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:28.958311 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.059792 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.059792 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059780 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059848 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059881 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059920 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.059951 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.060018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.060055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.060209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.060127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhsg5\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.061109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.060786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.061109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.061040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.061557 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.061513 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.061657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.061594 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.061732 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.061668 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.064127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.064055 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.070371 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.064620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.089873 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.089838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.108568 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.108536 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhsg5\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5\") pod \"e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.125530 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.125506 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:29.436204 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.436179 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:03:29.438016 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:29.437992 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd645f7b_02fd_44c7_abfe_483285abd5aa.slice/crio-8557cba974ec587e2ca65a24221a5ac34b3be2deca338a883efaf1f5e2cf6ab8 WatchSource:0}: Error finding container 8557cba974ec587e2ca65a24221a5ac34b3be2deca338a883efaf1f5e2cf6ab8: Status 404 returned error can't find the container with id 8557cba974ec587e2ca65a24221a5ac34b3be2deca338a883efaf1f5e2cf6ab8 Sep 08 17:03:29.440303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.440273 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.440442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.440358 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.440442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.440386 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.840058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.839973 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerStarted","Data":"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7"} Sep 08 17:03:29.840058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.840027 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerStarted","Data":"8557cba974ec587e2ca65a24221a5ac34b3be2deca338a883efaf1f5e2cf6ab8"} Sep 08 17:03:29.841607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.841584 2556 generic.go:356] "Generic (PLEG): container finished" podID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerID="a5f03abff40289b6d459af466fe7cf22357b9f24f13dc8f2ae5179c0d9007094" exitCode=0 Sep 08 17:03:29.841710 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.841652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerDied","Data":"a5f03abff40289b6d459af466fe7cf22357b9f24f13dc8f2ae5179c0d9007094"} Sep 08 17:03:29.842009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.841996 2556 scope.go:117] "RemoveContainer" containerID="a5f03abff40289b6d459af466fe7cf22357b9f24f13dc8f2ae5179c0d9007094" Sep 08 17:03:29.843292 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.843239 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.843413 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.843367 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.843477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.843418 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:29.881146 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:29.881097 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podStartSLOduration=1.8810804349999999 podStartE2EDuration="1.881080435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:29.880348789 +0000 UTC m=+741.025119347" watchObservedRunningTime="2026-09-08 17:03:29.881080435 +0000 UTC m=+741.025850996" Sep 08 17:03:30.126687 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:30.126642 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:30.131989 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:30.131965 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:30.856960 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:30.856925 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerStarted","Data":"707d2312b5d462baeea736ef020321e471bd87be6396b93d20616bbf0b306540"} Sep 08 17:03:30.857547 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:30.857525 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:30.864125 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:30.864093 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:31.055777 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:31.054345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:31.056127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:31.056102 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:31.057116 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:31.057084 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:32.053751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:32.053703 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:32.054264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:32.053775 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:33.053735 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:33.053689 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:33.053944 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:33.053760 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:34.054403 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.054363 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:34.054998 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.054966 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:34.571000 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.570959 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:03:34.575439 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.575344 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.586856 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.581247 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5a7cab-dockercfg-fhjbl\"" Sep 08 17:03:34.586856 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.581814 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a7cab\"" Sep 08 17:03:34.593266 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.593240 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:03:34.729608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.729568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7hp\" (UniqueName: \"kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.729770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.729639 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.830636 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.830543 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.830799 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.830692 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7hp\" (UniqueName: \"kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.831301 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.831280 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.854444 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.854407 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7hp\" (UniqueName: \"kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp\") pod \"payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:34.895273 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:34.895237 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:35.061347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:35.060789 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:35.061347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:35.060869 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:35.204406 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:35.204369 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:03:35.209155 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:35.209123 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7a3d4a_4a0d_41a4_a33a_8d0f87bc9eae.slice/crio-4e9165599a1295f7e5bedf573bb5842102a2197ebb07bdb513745e92e945bd89 WatchSource:0}: Error finding container 4e9165599a1295f7e5bedf573bb5842102a2197ebb07bdb513745e92e945bd89: Status 404 returned error can't find the container with id 4e9165599a1295f7e5bedf573bb5842102a2197ebb07bdb513745e92e945bd89 Sep 08 17:03:35.211160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:35.211142 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:03:35.888502 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:35.888467 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" event={"ID":"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae","Type":"ContainerStarted","Data":"4e9165599a1295f7e5bedf573bb5842102a2197ebb07bdb513745e92e945bd89"} Sep 08 17:03:36.055903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:36.055856 2556 patch_prober.go:28] interesting pod/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 17:03:36.056115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:36.055920 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 17:03:37.058473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.058444 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:37.058893 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.058868 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:37.059588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.059573 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:03:37.904417 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.904383 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" event={"ID":"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae","Type":"ContainerStarted","Data":"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984"} Sep 08 17:03:37.904629 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.904612 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:37.927776 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:37.927725 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" podStartSLOduration=1.458564992 podStartE2EDuration="3.927706545s" totalImagesPullingTime="2.469141553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:37.927020135 +0000 UTC m=+749.071790695" watchObservedRunningTime="2026-09-08 17:03:37.927706545 +0000 UTC m=+749.072477106" Sep 08 17:03:38.916735 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.916701 2556 generic.go:356] "Generic (PLEG): container finished" podID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerID="2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777" exitCode=0 Sep 08 17:03:38.917207 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.916783 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerDied","Data":"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777"} Sep 08 17:03:38.917397 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.917377 2556 scope.go:117] "RemoveContainer" containerID="2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777" Sep 08 17:03:38.918371 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.918346 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:38.918430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.918405 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:38.918464 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:38.918436 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:39.923469 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:39.923432 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerStarted","Data":"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658"} Sep 08 17:03:40.887223 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:40.887162 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:40.887548 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:40.887524 2556 patch_prober.go:28] interesting pod/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 08 17:03:40.887604 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:40.887574 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 08 17:03:41.854286 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.854247 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd"] Sep 08 17:03:41.858222 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.858191 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:41.874912 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.874874 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:03:41.875061 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.875003 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:03:41.885991 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.885960 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd"] Sep 08 17:03:41.887197 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.887174 2556 patch_prober.go:28] interesting pod/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 08 17:03:41.887585 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:41.887225 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 08 17:03:42.011555 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.011511 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd\" (UID: \"7eb55cea-c9c4-4342-a953-6b18fef9d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:42.112211 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.112126 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd\" (UID: \"7eb55cea-c9c4-4342-a953-6b18fef9d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:42.112800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.112778 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd\" (UID: \"7eb55cea-c9c4-4342-a953-6b18fef9d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:42.173630 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.173584 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:42.463073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.462982 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd"] Sep 08 17:03:42.469477 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:42.469448 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb55cea_c9c4_4342_a953_6b18fef9d11f.slice/crio-f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92 WatchSource:0}: Error finding container f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92: Status 404 returned error can't find the container with id f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92 Sep 08 17:03:42.866435 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.866399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:03:42.887674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.887637 2556 patch_prober.go:28] interesting pod/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 08 17:03:42.887829 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.887692 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 08 17:03:42.939809 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:42.939772 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" event={"ID":"7eb55cea-c9c4-4342-a953-6b18fef9d11f","Type":"ContainerStarted","Data":"f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92"} Sep 08 17:03:43.888157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:43.888116 2556 patch_prober.go:28] interesting pod/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 08 17:03:43.888672 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:43.888189 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 08 17:03:44.887849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:44.887802 2556 patch_prober.go:28] interesting pod/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Sep 08 17:03:44.888051 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:44.887899 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Sep 08 17:03:45.892025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:45.891996 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:45.892384 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:45.892259 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:45.892953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:45.892934 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:03:45.972281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:45.972245 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" event={"ID":"7eb55cea-c9c4-4342-a953-6b18fef9d11f","Type":"ContainerStarted","Data":"a5d3b70d1affa859f2fc62d3e98480d4c7303dfa1f6bc2a4e01fa21ead11390a"} Sep 08 17:03:46.033446 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:46.033397 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" podStartSLOduration=1.979063536 podStartE2EDuration="5.033381249s" totalImagesPullingTime="3.054317713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:46.031922979 +0000 UTC m=+757.176693539" watchObservedRunningTime="2026-09-08 17:03:46.033381249 +0000 UTC m=+757.178151810" Sep 08 17:03:46.978644 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:46.978609 2556 generic.go:356] "Generic (PLEG): container finished" podID="7eb55cea-c9c4-4342-a953-6b18fef9d11f" containerID="a5d3b70d1affa859f2fc62d3e98480d4c7303dfa1f6bc2a4e01fa21ead11390a" exitCode=0 Sep 08 17:03:46.979067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:46.978699 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" event={"ID":"7eb55cea-c9c4-4342-a953-6b18fef9d11f","Type":"ContainerDied","Data":"a5d3b70d1affa859f2fc62d3e98480d4c7303dfa1f6bc2a4e01fa21ead11390a"} Sep 08 17:03:48.138668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.138642 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:48.296118 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.296014 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca\") pod \"7eb55cea-c9c4-4342-a953-6b18fef9d11f\" (UID: \"7eb55cea-c9c4-4342-a953-6b18fef9d11f\") " Sep 08 17:03:48.296403 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.296379 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7eb55cea-c9c4-4342-a953-6b18fef9d11f" (UID: "7eb55cea-c9c4-4342-a953-6b18fef9d11f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:03:48.397072 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.397034 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb55cea-c9c4-4342-a953-6b18fef9d11f-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:03:48.919538 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.919508 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:03:48.991824 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.991796 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" Sep 08 17:03:48.992029 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.991812 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd" event={"ID":"7eb55cea-c9c4-4342-a953-6b18fef9d11f","Type":"ContainerDied","Data":"f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92"} Sep 08 17:03:48.992029 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:48.991872 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37e5326105cc47316ad8b0626db45863da235dfe23536b2241fdf06027dcb92" Sep 08 17:03:54.019029 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.018989 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerID="4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7" exitCode=0 Sep 08 17:03:54.019029 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.019045 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerDied","Data":"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7"} Sep 08 17:03:54.019704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.019636 2556 scope.go:117] "RemoveContainer" containerID="4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7" Sep 08 17:03:54.021578 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.021547 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:54.021717 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.021632 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:54.021717 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.021682 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:54.700694 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.700654 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd"] Sep 08 17:03:54.705074 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.705047 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a57edc2e-tvvpd"] Sep 08 17:03:54.905013 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.904973 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:03:54.905212 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:54.905192 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" containerID="cri-o://707d2312b5d462baeea736ef020321e471bd87be6396b93d20616bbf0b306540" gracePeriod=30 Sep 08 17:03:55.030914 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:55.029884 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerStarted","Data":"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b"} Sep 08 17:03:55.125701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:55.125662 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:03:55.126019 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:55.125994 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:03:55.126074 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:55.126049 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:03:55.342420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:55.342306 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb55cea-c9c4-4342-a953-6b18fef9d11f" path="/var/lib/kubelet/pods/7eb55cea-c9c4-4342-a953-6b18fef9d11f/volumes" Sep 08 17:03:56.126078 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.126043 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:03:56.126565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.126105 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:03:56.784194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.784157 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:03:56.784682 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.784663 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb55cea-c9c4-4342-a953-6b18fef9d11f" containerName="revoke-keys" Sep 08 17:03:56.784800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.784713 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb55cea-c9c4-4342-a953-6b18fef9d11f" containerName="revoke-keys" Sep 08 17:03:56.784886 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.784872 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb55cea-c9c4-4342-a953-6b18fef9d11f" containerName="revoke-keys" Sep 08 17:03:56.792230 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.792207 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.802822 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.802791 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-56569e47-gw-openshift-default-dockercfg-q846t\"" Sep 08 17:03:56.854574 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.854536 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:03:56.882799 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.882763 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.882984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.882807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.882984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.882832 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.882984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.882918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.883160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.883023 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.883160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.883059 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6s4\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.883160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.883100 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.883160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.883127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.883306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.883161 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984569 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6s4\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984596 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984634 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984688 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984731 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.984785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.984769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.985184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.985070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.985184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.985122 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.985184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.985154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.985369 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.985287 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.985645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.985621 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.987078 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.987051 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.987198 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.987179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:56.999786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:56.999761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:57.025715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.025679 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6s4\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4\") pod \"e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:57.102524 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.102433 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:57.126464 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.126429 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:03:57.126831 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.126489 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:03:57.451861 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.451812 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:03:57.457081 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:03:57.457044 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6dfe64_5231_4a84_9b90_a2c5bb2d2127.slice/crio-8292828edb4bc8a637c6ce53e2a2f3ccff1f72cbcdef71d0618b012bbc5a59dd WatchSource:0}: Error finding container 8292828edb4bc8a637c6ce53e2a2f3ccff1f72cbcdef71d0618b012bbc5a59dd: Status 404 returned error can't find the container with id 8292828edb4bc8a637c6ce53e2a2f3ccff1f72cbcdef71d0618b012bbc5a59dd Sep 08 17:03:57.460270 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.460210 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:57.460393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.460316 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:57.460393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:57.460386 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:03:58.053207 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.053162 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerStarted","Data":"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84"} Sep 08 17:03:58.053429 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.053269 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerStarted","Data":"8292828edb4bc8a637c6ce53e2a2f3ccff1f72cbcdef71d0618b012bbc5a59dd"} Sep 08 17:03:58.098790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.098741 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podStartSLOduration=2.098725745 podStartE2EDuration="2.098725745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:58.096570093 +0000 UTC m=+769.241340683" watchObservedRunningTime="2026-09-08 17:03:58.098725745 +0000 UTC m=+769.243496305" Sep 08 17:03:58.102794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.102767 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:58.104399 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.104375 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:03:58.104497 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.104426 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:03:58.126417 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.126382 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:03:58.126535 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:58.126443 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:03:59.107145 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:59.107111 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:03:59.126018 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:59.125985 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:03:59.126153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:03:59.126033 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:04:00.080430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.080379 2556 generic.go:356] "Generic (PLEG): container finished" podID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerID="707d2312b5d462baeea736ef020321e471bd87be6396b93d20616bbf0b306540" exitCode=0 Sep 08 17:04:00.081011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.080982 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerDied","Data":"707d2312b5d462baeea736ef020321e471bd87be6396b93d20616bbf0b306540"} Sep 08 17:04:00.081140 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.081056 2556 scope.go:117] "RemoveContainer" containerID="a5f03abff40289b6d459af466fe7cf22357b9f24f13dc8f2ae5179c0d9007094" Sep 08 17:04:00.081215 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.081198 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:00.082493 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.082377 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:00.126725 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.126696 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:04:00.126908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.126754 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:04:00.207607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.207578 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:04:00.321055 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321022 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2rfk\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321200 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321070 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321200 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321113 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321309 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321205 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321309 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321249 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321309 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321288 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321488 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321346 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321488 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321373 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321488 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321408 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert\") pod \"3d479400-7cd9-477d-af35-0380bdc7fd76\" (UID: \"3d479400-7cd9-477d-af35-0380bdc7fd76\") " Sep 08 17:04:00.321488 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321407 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:00.321715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321536 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:00.321715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321554 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:00.321715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321583 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:00.321883 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321839 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.321883 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321860 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.321883 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321876 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.322042 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321891 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.322042 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.321872 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:00.323627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.323601 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:00.323627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.323618 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:04:00.323786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.323620 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk" (OuterVolumeSpecName: "kube-api-access-h2rfk") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "kube-api-access-h2rfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:00.323786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.323741 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d479400-7cd9-477d-af35-0380bdc7fd76" (UID: "3d479400-7cd9-477d-af35-0380bdc7fd76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:00.422455 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.422421 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.422455 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.422450 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.422455 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.422458 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d479400-7cd9-477d-af35-0380bdc7fd76-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.422699 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.422469 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rfk\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-kube-api-access-h2rfk\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.422699 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.422481 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d479400-7cd9-477d-af35-0380bdc7fd76-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:00.857418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:00.857376 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:04:01.087252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.087218 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" Sep 08 17:04:01.087634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.087262 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz" event={"ID":"3d479400-7cd9-477d-af35-0380bdc7fd76","Type":"ContainerDied","Data":"0b014f9672f2b362ce90db28078a683edeb5a445fbe4c7701e33053f9ff92ac0"} Sep 08 17:04:01.087634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.087316 2556 scope.go:117] "RemoveContainer" containerID="707d2312b5d462baeea736ef020321e471bd87be6396b93d20616bbf0b306540" Sep 08 17:04:01.119192 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.119164 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:04:01.126469 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.126421 2556 patch_prober.go:28] interesting pod/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Sep 08 17:04:01.126999 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.126510 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Sep 08 17:04:01.128950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.128930 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a57edc2e-openshift-default-86f7995b88-b9jqz"] Sep 08 17:04:01.268783 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.268738 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:01.269314 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269291 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.269314 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269314 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.269514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269359 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.269514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269370 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.269514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269491 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.269514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.269507 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" containerName="istio-proxy" Sep 08 17:04:01.277122 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.277095 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.287106 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.287078 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-56569e47\"" Sep 08 17:04:01.287786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.287763 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-56569e47-dockercfg-894sm\"" Sep 08 17:04:01.288592 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.288569 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:01.293252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.293233 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.308635 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.308565 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:01.340740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.336449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.340740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.336545 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn89q\" (UniqueName: \"kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.340740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.336641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwpr\" (UniqueName: \"kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.340740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.336696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.356860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.356804 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d479400-7cd9-477d-af35-0380bdc7fd76" path="/var/lib/kubelet/pods/3d479400-7cd9-477d-af35-0380bdc7fd76/volumes" Sep 08 17:04:01.357770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.357735 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:01.437809 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.437702 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn89q\" (UniqueName: \"kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.437990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.437820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwpr\" (UniqueName: \"kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.437990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.437898 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.437990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.437963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.438700 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.438665 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.438842 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.438747 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.456204 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.456176 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn89q\" (UniqueName: \"kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q\") pod \"payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.457053 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.457030 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwpr\" (UniqueName: \"kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr\") pod \"payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.591032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.590994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:01.603278 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.603254 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:01.839568 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.839467 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:04:01.902371 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.902340 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:01.904343 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:04:01.904297 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccefd11_2b33_4af9_92a8_ca44b7b39719.slice/crio-50b4cf97e2e3e08187967b165f5a720a3549eef4d8e11dd03cf78247e41ee5f8 WatchSource:0}: Error finding container 50b4cf97e2e3e08187967b165f5a720a3549eef4d8e11dd03cf78247e41ee5f8: Status 404 returned error can't find the container with id 50b4cf97e2e3e08187967b165f5a720a3549eef4d8e11dd03cf78247e41ee5f8 Sep 08 17:04:01.918121 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:01.918096 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:01.921079 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:04:01.921052 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec310616_fb13_4161_b45b_e0d9b34f59c0.slice/crio-b0381c1c3f2abed5e8b036cae55daee60c9e7ec29db8bdec2aec639b931f2238 WatchSource:0}: Error finding container b0381c1c3f2abed5e8b036cae55daee60c9e7ec29db8bdec2aec639b931f2238: Status 404 returned error can't find the container with id b0381c1c3f2abed5e8b036cae55daee60c9e7ec29db8bdec2aec639b931f2238 Sep 08 17:04:02.096186 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.096080 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" event={"ID":"ec310616-fb13-4161-b45b-e0d9b34f59c0","Type":"ContainerStarted","Data":"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e"} Sep 08 17:04:02.096186 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.096144 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" event={"ID":"ec310616-fb13-4161-b45b-e0d9b34f59c0","Type":"ContainerStarted","Data":"b0381c1c3f2abed5e8b036cae55daee60c9e7ec29db8bdec2aec639b931f2238"} Sep 08 17:04:02.096677 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.096186 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:02.098302 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.098278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" event={"ID":"eccefd11-2b33-4af9-92a8-ca44b7b39719","Type":"ContainerStarted","Data":"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985"} Sep 08 17:04:02.098438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.098318 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" event={"ID":"eccefd11-2b33-4af9-92a8-ca44b7b39719","Type":"ContainerStarted","Data":"50b4cf97e2e3e08187967b165f5a720a3549eef4d8e11dd03cf78247e41ee5f8"} Sep 08 17:04:02.098528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.098515 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:02.130500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.130477 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:04:02.131385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.131363 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:04:02.138915 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.138874 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" podStartSLOduration=1.138861224 podStartE2EDuration="1.138861224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:02.138287426 +0000 UTC m=+773.283057988" watchObservedRunningTime="2026-09-08 17:04:02.138861224 +0000 UTC m=+773.283631784" Sep 08 17:04:02.189123 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:02.189075 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" podStartSLOduration=1.18905568 podStartE2EDuration="1.18905568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:02.186673317 +0000 UTC m=+773.331443879" watchObservedRunningTime="2026-09-08 17:04:02.18905568 +0000 UTC m=+773.333826243" Sep 08 17:04:13.105705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:13.105674 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:13.106259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:13.105741 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:18.937274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:18.937240 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:04:21.223141 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.223111 2556 generic.go:356] "Generic (PLEG): container finished" podID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerID="4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84" exitCode=0 Sep 08 17:04:21.223564 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.223159 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerDied","Data":"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84"} Sep 08 17:04:21.223654 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.223636 2556 scope.go:117] "RemoveContainer" containerID="4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84" Sep 08 17:04:21.224747 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.224708 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:21.224831 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.224817 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:21.224868 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:21.224852 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:22.230748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:22.230712 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerStarted","Data":"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a"} Sep 08 17:04:23.103477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:23.103435 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:23.103719 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:23.103695 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:04:23.103780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:23.103761 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:04:24.103813 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.103777 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:04:24.104203 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.103847 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:04:24.627793 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.627752 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf"] Sep 08 17:04:24.630257 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.630234 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:24.633065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.633036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:04:24.633432 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.633415 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:04:24.643476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.643451 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf"] Sep 08 17:04:24.697943 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.697910 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-56569e47-q26xf\" (UID: \"12a112c2-3d17-4d1a-b706-07bcedcbfa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:24.799092 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.799052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-56569e47-q26xf\" (UID: \"12a112c2-3d17-4d1a-b706-07bcedcbfa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:24.799743 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.799719 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-56569e47-q26xf\" (UID: \"12a112c2-3d17-4d1a-b706-07bcedcbfa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:24.940536 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:24.940440 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:25.103126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:25.103089 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:04:25.103304 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:25.103162 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:04:25.242114 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:25.242088 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf"] Sep 08 17:04:26.103350 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:26.103231 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:04:26.103506 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:26.103374 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:04:26.257665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:26.256801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" event={"ID":"12a112c2-3d17-4d1a-b706-07bcedcbfa5e","Type":"ContainerStarted","Data":"d5d7158d36e88cb267bf2fc039a7a122ede6efd0f88125e8abfa1f240b83370b"} Sep 08 17:04:26.257665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:26.256867 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" event={"ID":"12a112c2-3d17-4d1a-b706-07bcedcbfa5e","Type":"ContainerStarted","Data":"0fd8477cc31d0da2aa81551acca20c173e1f16fbd2dd9088b6c06f8ee833345f"} Sep 08 17:04:26.276867 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:26.276820 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" podStartSLOduration=2.276802685 podStartE2EDuration="2.276802685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:26.275160208 +0000 UTC m=+797.419930768" watchObservedRunningTime="2026-09-08 17:04:26.276802685 +0000 UTC m=+797.421573246" Sep 08 17:04:27.103009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:27.102974 2556 patch_prober.go:28] interesting pod/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Sep 08 17:04:27.103245 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:27.103222 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Sep 08 17:04:27.266536 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:27.266499 2556 generic.go:356] "Generic (PLEG): container finished" podID="12a112c2-3d17-4d1a-b706-07bcedcbfa5e" containerID="d5d7158d36e88cb267bf2fc039a7a122ede6efd0f88125e8abfa1f240b83370b" exitCode=0 Sep 08 17:04:27.267007 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:27.266619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" event={"ID":"12a112c2-3d17-4d1a-b706-07bcedcbfa5e","Type":"ContainerDied","Data":"d5d7158d36e88cb267bf2fc039a7a122ede6efd0f88125e8abfa1f240b83370b"} Sep 08 17:04:28.108290 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.108258 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:28.108657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.108635 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:28.109266 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.109244 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:28.424752 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.424730 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:28.540172 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.540134 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca\") pod \"12a112c2-3d17-4d1a-b706-07bcedcbfa5e\" (UID: \"12a112c2-3d17-4d1a-b706-07bcedcbfa5e\") " Sep 08 17:04:28.540516 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.540491 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12a112c2-3d17-4d1a-b706-07bcedcbfa5e" (UID: "12a112c2-3d17-4d1a-b706-07bcedcbfa5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:28.641986 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:28.641945 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a112c2-3d17-4d1a-b706-07bcedcbfa5e-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:29.279015 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:29.278981 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" event={"ID":"12a112c2-3d17-4d1a-b706-07bcedcbfa5e","Type":"ContainerDied","Data":"0fd8477cc31d0da2aa81551acca20c173e1f16fbd2dd9088b6c06f8ee833345f"} Sep 08 17:04:29.279015 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:29.279003 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf" Sep 08 17:04:29.279233 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:29.279034 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd8477cc31d0da2aa81551acca20c173e1f16fbd2dd9088b6c06f8ee833345f" Sep 08 17:04:31.600583 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.600547 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:31.601117 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.600801 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" podUID="ec310616-fb13-4161-b45b-e0d9b34f59c0" containerName="payload-processing" containerID="cri-o://3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e" gracePeriod=30 Sep 08 17:04:31.608732 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.608706 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:31.608926 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.608902 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" podUID="eccefd11-2b33-4af9-92a8-ca44b7b39719" containerName="payload-pre-processing" containerID="cri-o://c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985" gracePeriod=30 Sep 08 17:04:31.858834 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.858341 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf"] Sep 08 17:04:31.862423 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.862365 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-56569e47-q26xf"] Sep 08 17:04:31.891181 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.891156 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:31.931667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.931644 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:31.975286 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975255 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume\") pod \"ec310616-fb13-4161-b45b-e0d9b34f59c0\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " Sep 08 17:04:31.975484 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975367 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctwpr\" (UniqueName: \"kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr\") pod \"ec310616-fb13-4161-b45b-e0d9b34f59c0\" (UID: \"ec310616-fb13-4161-b45b-e0d9b34f59c0\") " Sep 08 17:04:31.975484 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975392 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn89q\" (UniqueName: \"kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q\") pod \"eccefd11-2b33-4af9-92a8-ca44b7b39719\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " Sep 08 17:04:31.975484 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975422 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume\") pod \"eccefd11-2b33-4af9-92a8-ca44b7b39719\" (UID: \"eccefd11-2b33-4af9-92a8-ca44b7b39719\") " Sep 08 17:04:31.975821 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975790 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec310616-fb13-4161-b45b-e0d9b34f59c0" (UID: "ec310616-fb13-4161-b45b-e0d9b34f59c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:31.975907 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.975832 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eccefd11-2b33-4af9-92a8-ca44b7b39719" (UID: "eccefd11-2b33-4af9-92a8-ca44b7b39719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:31.977538 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.977512 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr" (OuterVolumeSpecName: "kube-api-access-ctwpr") pod "ec310616-fb13-4161-b45b-e0d9b34f59c0" (UID: "ec310616-fb13-4161-b45b-e0d9b34f59c0"). InnerVolumeSpecName "kube-api-access-ctwpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:31.977610 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:31.977531 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q" (OuterVolumeSpecName: "kube-api-access-qn89q") pod "eccefd11-2b33-4af9-92a8-ca44b7b39719" (UID: "eccefd11-2b33-4af9-92a8-ca44b7b39719"). InnerVolumeSpecName "kube-api-access-qn89q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:32.032168 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.032136 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:04:32.032443 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.032418 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" containerID="cri-o://5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a" gracePeriod=30 Sep 08 17:04:32.076711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.076676 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccefd11-2b33-4af9-92a8-ca44b7b39719-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:32.076711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.076704 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec310616-fb13-4161-b45b-e0d9b34f59c0-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:32.076711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.076713 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctwpr\" (UniqueName: \"kubernetes.io/projected/ec310616-fb13-4161-b45b-e0d9b34f59c0-kube-api-access-ctwpr\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:32.076944 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.076721 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn89q\" (UniqueName: \"kubernetes.io/projected/eccefd11-2b33-4af9-92a8-ca44b7b39719-kube-api-access-qn89q\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:32.295614 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.295583 2556 generic.go:356] "Generic (PLEG): container finished" podID="ec310616-fb13-4161-b45b-e0d9b34f59c0" containerID="3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e" exitCode=0 Sep 08 17:04:32.295802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.295646 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" Sep 08 17:04:32.295802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.295666 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" event={"ID":"ec310616-fb13-4161-b45b-e0d9b34f59c0","Type":"ContainerDied","Data":"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e"} Sep 08 17:04:32.295802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.295721 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d" event={"ID":"ec310616-fb13-4161-b45b-e0d9b34f59c0","Type":"ContainerDied","Data":"b0381c1c3f2abed5e8b036cae55daee60c9e7ec29db8bdec2aec639b931f2238"} Sep 08 17:04:32.295802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.295753 2556 scope.go:117] "RemoveContainer" containerID="3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e" Sep 08 17:04:32.296995 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.296975 2556 generic.go:356] "Generic (PLEG): container finished" podID="eccefd11-2b33-4af9-92a8-ca44b7b39719" containerID="c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985" exitCode=0 Sep 08 17:04:32.297084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.297026 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" Sep 08 17:04:32.297084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.297037 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" event={"ID":"eccefd11-2b33-4af9-92a8-ca44b7b39719","Type":"ContainerDied","Data":"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985"} Sep 08 17:04:32.297084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.297068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h" event={"ID":"eccefd11-2b33-4af9-92a8-ca44b7b39719","Type":"ContainerDied","Data":"50b4cf97e2e3e08187967b165f5a720a3549eef4d8e11dd03cf78247e41ee5f8"} Sep 08 17:04:32.305162 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.305130 2556 scope.go:117] "RemoveContainer" containerID="3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e" Sep 08 17:04:32.305483 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:04:32.305454 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e\": container with ID starting with 3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e not found: ID does not exist" containerID="3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e" Sep 08 17:04:32.305601 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.305494 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e"} err="failed to get container status \"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e\": rpc error: code = NotFound desc = could not find container \"3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e\": container with ID starting with 3953f0220b064bb3352879da6eb2556db31e8efe10fc82e4da02b5c84c07dc6e not found: ID does not exist" Sep 08 17:04:32.305601 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.305518 2556 scope.go:117] "RemoveContainer" containerID="c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985" Sep 08 17:04:32.313293 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.313277 2556 scope.go:117] "RemoveContainer" containerID="c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985" Sep 08 17:04:32.313580 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:04:32.313552 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985\": container with ID starting with c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985 not found: ID does not exist" containerID="c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985" Sep 08 17:04:32.313660 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.313587 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985"} err="failed to get container status \"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985\": rpc error: code = NotFound desc = could not find container \"c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985\": container with ID starting with c641cdddc4916ffa92c55e7aab2d02e0be3f2b30fd428d0f1ae4d29e41061985 not found: ID does not exist" Sep 08 17:04:32.321989 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.321967 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:32.326001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.325979 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-56569e47-7454fc576b-p8g5d"] Sep 08 17:04:32.336496 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.336474 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:32.341460 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.341436 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-56569e47-6c7568f8db-d9n4h"] Sep 08 17:04:32.735624 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.735591 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:04:32.736138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736107 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec310616-fb13-4161-b45b-e0d9b34f59c0" containerName="payload-processing" Sep 08 17:04:32.736138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736133 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec310616-fb13-4161-b45b-e0d9b34f59c0" containerName="payload-processing" Sep 08 17:04:32.736289 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736189 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a112c2-3d17-4d1a-b706-07bcedcbfa5e" containerName="revoke-keys" Sep 08 17:04:32.736289 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736200 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a112c2-3d17-4d1a-b706-07bcedcbfa5e" containerName="revoke-keys" Sep 08 17:04:32.736289 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736216 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eccefd11-2b33-4af9-92a8-ca44b7b39719" containerName="payload-pre-processing" Sep 08 17:04:32.736289 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736225 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="eccefd11-2b33-4af9-92a8-ca44b7b39719" containerName="payload-pre-processing" Sep 08 17:04:32.736442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736335 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="eccefd11-2b33-4af9-92a8-ca44b7b39719" containerName="payload-pre-processing" Sep 08 17:04:32.736442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736348 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a112c2-3d17-4d1a-b706-07bcedcbfa5e" containerName="revoke-keys" Sep 08 17:04:32.736442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.736358 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec310616-fb13-4161-b45b-e0d9b34f59c0" containerName="payload-processing" Sep 08 17:04:32.740993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.740975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.744258 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.744238 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f155bce9-openshift-default-dockercfg-89vfz\"" Sep 08 17:04:32.764704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.764679 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:04:32.782810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.782983 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782815 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.782983 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.782983 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782930 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.782983 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782968 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.783210 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.782987 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.783210 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.783019 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.783210 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.783098 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.783397 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.783229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkdc\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885268 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885268 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885344 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885454 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkdc\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885496 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885573 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.885812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.885613 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.886344 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.886044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.886344 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.886290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.887848 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.886533 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.887848 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.887288 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.887848 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.887633 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.892950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.890857 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.892950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.892883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.903208 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.900872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkdc\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:32.903208 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:32.903117 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token\") pod \"e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:33.051506 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.051469 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:33.335705 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:04:33.335651 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8e901c_5a91_41f9_b2c7_502d4a36db6d.slice/crio-9b9e523ad331ee93a3853c8c98b38363573965341f8277894d857fe0142aafe0 WatchSource:0}: Error finding container 9b9e523ad331ee93a3853c8c98b38363573965341f8277894d857fe0142aafe0: Status 404 returned error can't find the container with id 9b9e523ad331ee93a3853c8c98b38363573965341f8277894d857fe0142aafe0 Sep 08 17:04:33.338097 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.338057 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:33.338224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.338152 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:33.338224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.338200 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:33.342955 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.342925 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a112c2-3d17-4d1a-b706-07bcedcbfa5e" path="/var/lib/kubelet/pods/12a112c2-3d17-4d1a-b706-07bcedcbfa5e/volumes" Sep 08 17:04:33.343288 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.343273 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec310616-fb13-4161-b45b-e0d9b34f59c0" path="/var/lib/kubelet/pods/ec310616-fb13-4161-b45b-e0d9b34f59c0/volumes" Sep 08 17:04:33.343785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.343730 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eccefd11-2b33-4af9-92a8-ca44b7b39719" path="/var/lib/kubelet/pods/eccefd11-2b33-4af9-92a8-ca44b7b39719/volumes" Sep 08 17:04:33.344043 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:33.343971 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:04:34.312443 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:34.312411 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerStarted","Data":"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a"} Sep 08 17:04:34.312829 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:34.312458 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerStarted","Data":"9b9e523ad331ee93a3853c8c98b38363573965341f8277894d857fe0142aafe0"} Sep 08 17:04:34.326636 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:34.326601 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:04:34.341183 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:34.341134 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podStartSLOduration=2.341118819 podStartE2EDuration="2.341118819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:34.339190396 +0000 UTC m=+805.483960959" watchObservedRunningTime="2026-09-08 17:04:34.341118819 +0000 UTC m=+805.485889381" Sep 08 17:04:35.051797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:35.051761 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:35.056723 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:35.056700 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:35.318231 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:35.318144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:35.319186 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:35.319164 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:37.286663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.286640 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:37.329518 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.329448 2556 generic.go:356] "Generic (PLEG): container finished" podID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerID="5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a" exitCode=0 Sep 08 17:04:37.329518 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.329496 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" Sep 08 17:04:37.329685 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.329532 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerDied","Data":"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a"} Sep 08 17:04:37.329685 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.329587 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd" event={"ID":"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127","Type":"ContainerDied","Data":"8292828edb4bc8a637c6ce53e2a2f3ccff1f72cbcdef71d0618b012bbc5a59dd"} Sep 08 17:04:37.329685 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.329615 2556 scope.go:117] "RemoveContainer" containerID="5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a" Sep 08 17:04:37.330115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330094 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330130 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330163 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330239 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh6s4\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330280 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330311 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330389 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330625 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330418 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:37.330625 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330451 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330625 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330435 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:37.330625 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330500 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy\") pod \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\" (UID: \"ca6dfe64-5231-4a84-9b90-a2c5bb2d2127\") " Sep 08 17:04:37.330833 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330640 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:37.330833 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330694 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:04:37.330833 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330719 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:37.330990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330839 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.330990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330857 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.330990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330866 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.330990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330876 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.330990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.330890 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.333070 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.333042 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4" (OuterVolumeSpecName: "kube-api-access-zh6s4") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "kube-api-access-zh6s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:37.333175 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.333046 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:04:37.333290 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.333242 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:04:37.333375 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.333345 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" (UID: "ca6dfe64-5231-4a84-9b90-a2c5bb2d2127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:04:37.355608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.355457 2556 scope.go:117] "RemoveContainer" containerID="4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84" Sep 08 17:04:37.364227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.364211 2556 scope.go:117] "RemoveContainer" containerID="5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a" Sep 08 17:04:37.364512 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:04:37.364494 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a\": container with ID starting with 5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a not found: ID does not exist" containerID="5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a" Sep 08 17:04:37.364581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.364518 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a"} err="failed to get container status \"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a\": rpc error: code = NotFound desc = could not find container \"5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a\": container with ID starting with 5ea4167ec082b991f0bb68d098285e7ca12f0cd78dce93b0d6136eb8a3cad95a not found: ID does not exist" Sep 08 17:04:37.364581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.364535 2556 scope.go:117] "RemoveContainer" containerID="4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84" Sep 08 17:04:37.364789 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:04:37.364769 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84\": container with ID starting with 4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84 not found: ID does not exist" containerID="4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84" Sep 08 17:04:37.364828 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.364795 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84"} err="failed to get container status \"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84\": rpc error: code = NotFound desc = could not find container \"4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84\": container with ID starting with 4b38f4488e7cf97560644eaddd5fe8eda2680c502410a63c6483d28d0db73b84 not found: ID does not exist" Sep 08 17:04:37.432412 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.432382 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.432412 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.432411 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh6s4\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-kube-api-access-zh6s4\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.432562 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.432423 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.432562 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.432436 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:04:37.653110 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.653076 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:04:37.659802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.659763 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-56569e47-gw-openshift-default-f89979d7b-pt5bd"] Sep 08 17:04:37.769433 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769400 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:04:37.769810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769798 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.769864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769811 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.769864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769821 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.769864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769826 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.769953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769892 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.769953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.769900 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" containerName="istio-proxy" Sep 08 17:04:37.774474 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.774456 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.777650 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.777629 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f155bce9\"" Sep 08 17:04:37.786640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.786616 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:04:37.836641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.836609 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.836825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.836786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmhb\" (UniqueName: \"kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.848265 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.848239 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:04:37.851823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.851808 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:37.855100 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.855083 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f155bce9-dockercfg-lsfmm\"" Sep 08 17:04:37.907380 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.907291 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:04:37.938128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.938095 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmhb\" (UniqueName: \"kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.938279 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.938154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.938279 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.938189 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slnv\" (UniqueName: \"kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:37.938279 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.938257 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:37.938860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.938841 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:37.959928 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:37.959900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmhb\" (UniqueName: \"kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb\") pod \"payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:38.039335 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.039285 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slnv\" (UniqueName: \"kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:38.039485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.039429 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:38.040113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.040092 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:38.050885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.050861 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slnv\" (UniqueName: \"kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv\") pod \"payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:38.085816 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.085787 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:38.161913 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.161829 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:38.370722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.370689 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:04:38.377071 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:04:38.377046 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f88e62_ef94_429e_b39d_c37d8ab4dced.slice/crio-529da500f8f3fc095d80c602a67d11bf9d8143d19e278d8c7e516c348629b304 WatchSource:0}: Error finding container 529da500f8f3fc095d80c602a67d11bf9d8143d19e278d8c7e516c348629b304: Status 404 returned error can't find the container with id 529da500f8f3fc095d80c602a67d11bf9d8143d19e278d8c7e516c348629b304 Sep 08 17:04:38.442395 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:38.442374 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:04:38.444644 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:04:38.444617 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62c6ae7_ae19_4043_b65a_e8f362e18926.slice/crio-341c77d15b6094e018bfc31fd3651921b7c0a51b89d5b47bcd792a24e95ee7d2 WatchSource:0}: Error finding container 341c77d15b6094e018bfc31fd3651921b7c0a51b89d5b47bcd792a24e95ee7d2: Status 404 returned error can't find the container with id 341c77d15b6094e018bfc31fd3651921b7c0a51b89d5b47bcd792a24e95ee7d2 Sep 08 17:04:39.353040 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353008 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca6dfe64-5231-4a84-9b90-a2c5bb2d2127" path="/var/lib/kubelet/pods/ca6dfe64-5231-4a84-9b90-a2c5bb2d2127/volumes" Sep 08 17:04:39.353292 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353275 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" event={"ID":"c62c6ae7-ae19-4043-b65a-e8f362e18926","Type":"ContainerStarted","Data":"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7"} Sep 08 17:04:39.353395 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353332 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" event={"ID":"c62c6ae7-ae19-4043-b65a-e8f362e18926","Type":"ContainerStarted","Data":"341c77d15b6094e018bfc31fd3651921b7c0a51b89d5b47bcd792a24e95ee7d2"} Sep 08 17:04:39.353395 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353354 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" event={"ID":"50f88e62-ef94-429e-b39d-c37d8ab4dced","Type":"ContainerStarted","Data":"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0"} Sep 08 17:04:39.353395 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353372 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:39.353395 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353391 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:39.353588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.353409 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" event={"ID":"50f88e62-ef94-429e-b39d-c37d8ab4dced","Type":"ContainerStarted","Data":"529da500f8f3fc095d80c602a67d11bf9d8143d19e278d8c7e516c348629b304"} Sep 08 17:04:39.370746 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.370692 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" podStartSLOduration=2.370675618 podStartE2EDuration="2.370675618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:39.368638019 +0000 UTC m=+810.513408581" watchObservedRunningTime="2026-09-08 17:04:39.370675618 +0000 UTC m=+810.515446174" Sep 08 17:04:39.388769 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:39.388710 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" podStartSLOduration=2.388690469 podStartE2EDuration="2.388690469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:39.38693336 +0000 UTC m=+810.531703932" watchObservedRunningTime="2026-09-08 17:04:39.388690469 +0000 UTC m=+810.533461031" Sep 08 17:04:50.361538 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:50.361504 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:04:50.362502 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:50.362474 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:04:52.430252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:52.430218 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:04:54.448465 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.448436 2556 generic.go:356] "Generic (PLEG): container finished" podID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerID="fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a" exitCode=0 Sep 08 17:04:54.448874 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.448471 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerDied","Data":"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a"} Sep 08 17:04:54.448874 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.448830 2556 scope.go:117] "RemoveContainer" containerID="fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a" Sep 08 17:04:54.449864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.449839 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:54.449924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.449910 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:54.449962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:54.449947 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:04:55.457118 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:55.457078 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerStarted","Data":"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad"} Sep 08 17:04:56.052084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:56.052038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:04:56.052470 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:56.052439 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:04:56.052572 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:56.052509 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:04:57.051966 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:57.051920 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:04:57.052495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:57.051997 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:04:58.052749 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:58.052703 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:04:58.053259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:58.052778 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:04:59.053158 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:59.053111 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:04:59.053672 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:04:59.053183 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:05:00.052783 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:00.052732 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:05:00.053056 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:00.052813 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:05:01.052125 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:01.052088 2556 patch_prober.go:28] interesting pod/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 08 17:05:01.052567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:01.052146 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 08 17:05:02.056283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:02.056252 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:05:02.056712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:02.056541 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:05:02.057247 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:02.057232 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:05:07.700061 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.700022 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:05:07.700665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.700282 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" podUID="50f88e62-ef94-429e-b39d-c37d8ab4dced" containerName="payload-processing" containerID="cri-o://074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0" gracePeriod=30 Sep 08 17:05:07.715724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.715694 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:05:07.715922 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.715900 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" podUID="c62c6ae7-ae19-4043-b65a-e8f362e18926" containerName="payload-pre-processing" containerID="cri-o://9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7" gracePeriod=30 Sep 08 17:05:07.963711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.963651 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:05:07.995368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:07.995317 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:05:08.087879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.087841 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2slnv\" (UniqueName: \"kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv\") pod \"c62c6ae7-ae19-4043-b65a-e8f362e18926\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " Sep 08 17:05:08.088081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.087927 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntmhb\" (UniqueName: \"kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb\") pod \"50f88e62-ef94-429e-b39d-c37d8ab4dced\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " Sep 08 17:05:08.088081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.087992 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume\") pod \"c62c6ae7-ae19-4043-b65a-e8f362e18926\" (UID: \"c62c6ae7-ae19-4043-b65a-e8f362e18926\") " Sep 08 17:05:08.088081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.088056 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume\") pod \"50f88e62-ef94-429e-b39d-c37d8ab4dced\" (UID: \"50f88e62-ef94-429e-b39d-c37d8ab4dced\") " Sep 08 17:05:08.088422 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.088391 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c62c6ae7-ae19-4043-b65a-e8f362e18926" (UID: "c62c6ae7-ae19-4043-b65a-e8f362e18926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:08.088685 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.088455 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50f88e62-ef94-429e-b39d-c37d8ab4dced" (UID: "50f88e62-ef94-429e-b39d-c37d8ab4dced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:08.088685 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.088509 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62c6ae7-ae19-4043-b65a-e8f362e18926-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.090137 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.090113 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb" (OuterVolumeSpecName: "kube-api-access-ntmhb") pod "50f88e62-ef94-429e-b39d-c37d8ab4dced" (UID: "50f88e62-ef94-429e-b39d-c37d8ab4dced"). InnerVolumeSpecName "kube-api-access-ntmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:08.090238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.090168 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv" (OuterVolumeSpecName: "kube-api-access-2slnv") pod "c62c6ae7-ae19-4043-b65a-e8f362e18926" (UID: "c62c6ae7-ae19-4043-b65a-e8f362e18926"). InnerVolumeSpecName "kube-api-access-2slnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:08.189160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.189127 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntmhb\" (UniqueName: \"kubernetes.io/projected/50f88e62-ef94-429e-b39d-c37d8ab4dced-kube-api-access-ntmhb\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.189160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.189154 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f88e62-ef94-429e-b39d-c37d8ab4dced-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.189160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.189163 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2slnv\" (UniqueName: \"kubernetes.io/projected/c62c6ae7-ae19-4043-b65a-e8f362e18926-kube-api-access-2slnv\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:08.540563 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.540526 2556 generic.go:356] "Generic (PLEG): container finished" podID="c62c6ae7-ae19-4043-b65a-e8f362e18926" containerID="9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7" exitCode=0 Sep 08 17:05:08.540773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.540584 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" Sep 08 17:05:08.540773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.540612 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" event={"ID":"c62c6ae7-ae19-4043-b65a-e8f362e18926","Type":"ContainerDied","Data":"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7"} Sep 08 17:05:08.540773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.540677 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7" event={"ID":"c62c6ae7-ae19-4043-b65a-e8f362e18926","Type":"ContainerDied","Data":"341c77d15b6094e018bfc31fd3651921b7c0a51b89d5b47bcd792a24e95ee7d2"} Sep 08 17:05:08.540773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.540709 2556 scope.go:117] "RemoveContainer" containerID="9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7" Sep 08 17:05:08.542016 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.541997 2556 generic.go:356] "Generic (PLEG): container finished" podID="50f88e62-ef94-429e-b39d-c37d8ab4dced" containerID="074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0" exitCode=0 Sep 08 17:05:08.542115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.542057 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" Sep 08 17:05:08.542115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.542077 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" event={"ID":"50f88e62-ef94-429e-b39d-c37d8ab4dced","Type":"ContainerDied","Data":"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0"} Sep 08 17:05:08.542115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.542111 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x" event={"ID":"50f88e62-ef94-429e-b39d-c37d8ab4dced","Type":"ContainerDied","Data":"529da500f8f3fc095d80c602a67d11bf9d8143d19e278d8c7e516c348629b304"} Sep 08 17:05:08.550075 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.550057 2556 scope.go:117] "RemoveContainer" containerID="9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7" Sep 08 17:05:08.550419 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:08.550392 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7\": container with ID starting with 9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7 not found: ID does not exist" containerID="9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7" Sep 08 17:05:08.550522 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.550430 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7"} err="failed to get container status \"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7\": rpc error: code = NotFound desc = could not find container \"9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7\": container with ID starting with 9b3a355b289cb5e3546672a2ffcbd20b60cfa2910a686095b220b21fd0fae4e7 not found: ID does not exist" Sep 08 17:05:08.550522 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.550456 2556 scope.go:117] "RemoveContainer" containerID="074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0" Sep 08 17:05:08.562213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.562194 2556 scope.go:117] "RemoveContainer" containerID="074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0" Sep 08 17:05:08.562488 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:08.562469 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0\": container with ID starting with 074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0 not found: ID does not exist" containerID="074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0" Sep 08 17:05:08.562543 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.562493 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0"} err="failed to get container status \"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0\": rpc error: code = NotFound desc = could not find container \"074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0\": container with ID starting with 074cddd804dde69320af1922b50fc10b5f78bc469450fdcc613fa683d50cd4f0 not found: ID does not exist" Sep 08 17:05:08.576398 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.576370 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:05:08.579581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.579557 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f155bce9-c46bd595-2kg6x"] Sep 08 17:05:08.592878 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.592853 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:05:08.599994 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:08.599963 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f155bce9-9f44ff7c8-zb7w7"] Sep 08 17:05:09.127063 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.127018 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:05:09.127524 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.127354 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" containerID="cri-o://fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad" gracePeriod=30 Sep 08 17:05:09.342810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.342779 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f88e62-ef94-429e-b39d-c37d8ab4dced" path="/var/lib/kubelet/pods/50f88e62-ef94-429e-b39d-c37d8ab4dced/volumes" Sep 08 17:05:09.343281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.343262 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62c6ae7-ae19-4043-b65a-e8f362e18926" path="/var/lib/kubelet/pods/c62c6ae7-ae19-4043-b65a-e8f362e18926/volumes" Sep 08 17:05:09.864190 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864157 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:09.864627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864613 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f88e62-ef94-429e-b39d-c37d8ab4dced" containerName="payload-processing" Sep 08 17:05:09.864690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864629 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f88e62-ef94-429e-b39d-c37d8ab4dced" containerName="payload-processing" Sep 08 17:05:09.864690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864648 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62c6ae7-ae19-4043-b65a-e8f362e18926" containerName="payload-pre-processing" Sep 08 17:05:09.864690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864671 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62c6ae7-ae19-4043-b65a-e8f362e18926" containerName="payload-pre-processing" Sep 08 17:05:09.864800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864751 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f88e62-ef94-429e-b39d-c37d8ab4dced" containerName="payload-processing" Sep 08 17:05:09.864800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.864761 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62c6ae7-ae19-4043-b65a-e8f362e18926" containerName="payload-pre-processing" Sep 08 17:05:09.869062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.869044 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:09.872049 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.872025 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b709328f-openshift-default-dockercfg-fzbr2\"" Sep 08 17:05:09.892886 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:09.892860 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:10.009368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009316 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009371 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009398 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009556 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwvv\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009577 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.009784 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.009592 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110237 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110258 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110283 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110449 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110673 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110547 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110673 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwvv\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110673 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110631 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110673 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110660 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110683 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.110872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110715 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.111011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.110988 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.111077 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.111019 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.111133 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.111111 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.112681 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.112659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.112973 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.112956 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.125012 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.124952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwvv\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.129824 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.129803 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token\") pod \"e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.180297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.180253 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:10.464746 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.464717 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:10.465878 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:05:10.465853 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11350a39_3813_4057_926f_de83628927e4.slice/crio-faafac4b15a9403485c2ca7997d50355e68db6c72f52efc337245f4067ef1374 WatchSource:0}: Error finding container faafac4b15a9403485c2ca7997d50355e68db6c72f52efc337245f4067ef1374: Status 404 returned error can't find the container with id faafac4b15a9403485c2ca7997d50355e68db6c72f52efc337245f4067ef1374 Sep 08 17:05:10.468091 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.468059 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:10.468151 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.468127 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:10.468189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.468156 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:10.554064 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.554030 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerStarted","Data":"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819"} Sep 08 17:05:10.554220 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.554093 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerStarted","Data":"faafac4b15a9403485c2ca7997d50355e68db6c72f52efc337245f4067ef1374"} Sep 08 17:05:10.581239 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:10.581173 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podStartSLOduration=1.5811513050000001 podStartE2EDuration="1.581151305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:10.58059305 +0000 UTC m=+841.725363611" watchObservedRunningTime="2026-09-08 17:05:10.581151305 +0000 UTC m=+841.725921863" Sep 08 17:05:11.181259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:11.181221 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:11.186474 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:11.186450 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:11.559507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:11.559419 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:11.560359 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:11.560339 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:12.432563 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:12.432530 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:05:14.053057 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.053020 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:14.057035 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.057014 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.059996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.059973 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b709328f-dockercfg-dttd9\"" Sep 08 17:05:14.060209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.060196 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b709328f\"" Sep 08 17:05:14.070676 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.070657 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:14.119595 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.119564 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:14.123142 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.123125 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.133151 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.133126 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:14.152784 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.152754 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz55t\" (UniqueName: \"kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.152887 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.152871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.254479 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.254422 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz55t\" (UniqueName: \"kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.254617 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.254577 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.254736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.254719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.254779 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.254753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccjd\" (UniqueName: \"kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.255510 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.255486 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.263586 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.263555 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz55t\" (UniqueName: \"kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t\") pod \"payload-processing-e2e-derive-b709328f-589c9f566f-ktt56\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.356365 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.356247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.356365 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.356288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccjd\" (UniqueName: \"kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.356921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.356897 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.368110 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.368080 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:14.370872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.370848 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccjd\" (UniqueName: \"kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd\") pod \"payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.397558 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.397524 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:05:14.434623 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.434526 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:14.557839 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.557808 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlkdc\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.557849 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.557875 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.557904 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.557968 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558175 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558013 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558175 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558071 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558175 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558112 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558175 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558167 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token\") pod \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\" (UID: \"6a8e901c-5a91-41f9-b2c7-502d4a36db6d\") " Sep 08 17:05:14.558459 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558399 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:14.558885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558643 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.558885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558796 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:14.558885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558841 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:14.558885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:14.558885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.558869 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:14.560551 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.560516 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc" (OuterVolumeSpecName: "kube-api-access-wlkdc") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "kube-api-access-wlkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:14.560695 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.560671 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:14.560896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.560876 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:05:14.560990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.560879 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a8e901c-5a91-41f9-b2c7-502d4a36db6d" (UID: "6a8e901c-5a91-41f9-b2c7-502d4a36db6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:14.575370 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.575339 2556 generic.go:356] "Generic (PLEG): container finished" podID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerID="fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad" exitCode=0 Sep 08 17:05:14.575478 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.575390 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" Sep 08 17:05:14.575478 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.575400 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerDied","Data":"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad"} Sep 08 17:05:14.575478 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.575453 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px" event={"ID":"6a8e901c-5a91-41f9-b2c7-502d4a36db6d","Type":"ContainerDied","Data":"9b9e523ad331ee93a3853c8c98b38363573965341f8277894d857fe0142aafe0"} Sep 08 17:05:14.575595 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.575485 2556 scope.go:117] "RemoveContainer" containerID="fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad" Sep 08 17:05:14.587566 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.587546 2556 scope.go:117] "RemoveContainer" containerID="fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a" Sep 08 17:05:14.595761 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.595736 2556 scope.go:117] "RemoveContainer" containerID="fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad" Sep 08 17:05:14.596078 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:14.596058 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad\": container with ID starting with fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad not found: ID does not exist" containerID="fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad" Sep 08 17:05:14.596164 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.596092 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad"} err="failed to get container status \"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad\": rpc error: code = NotFound desc = could not find container \"fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad\": container with ID starting with fd2c04be69020b0ad2707bb2ffb7286cadd34044e361cf1a3de647e86484b0ad not found: ID does not exist" Sep 08 17:05:14.596164 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.596119 2556 scope.go:117] "RemoveContainer" containerID="fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a" Sep 08 17:05:14.596405 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:14.596385 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a\": container with ID starting with fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a not found: ID does not exist" containerID="fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a" Sep 08 17:05:14.596456 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.596411 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a"} err="failed to get container status \"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a\": rpc error: code = NotFound desc = could not find container \"fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a\": container with ID starting with fe7d9c064259b8d4083c25f30d22e9c5be53d6a1e5f8d250ec26ab5d8cbf3b7a not found: ID does not exist" Sep 08 17:05:14.599941 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.599919 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:05:14.603657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.603636 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f155bce9-openshift-default-99d9f7888-qx4px"] Sep 08 17:05:14.644975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.644946 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:14.647734 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:05:14.647705 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4242b55_9db4_431c_999c_59a91d492a58.slice/crio-fabf9e79ea06152d9b4a7bea67537ad42d72154044bdc0c97ee3223f049e8338 WatchSource:0}: Error finding container fabf9e79ea06152d9b4a7bea67537ad42d72154044bdc0c97ee3223f049e8338: Status 404 returned error can't find the container with id fabf9e79ea06152d9b4a7bea67537ad42d72154044bdc0c97ee3223f049e8338 Sep 08 17:05:14.659857 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659834 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkdc\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-kube-api-access-wlkdc\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659858 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659872 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659887 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659899 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659911 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659924 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.659963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.659936 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e901c-5a91-41f9-b2c7-502d4a36db6d-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:14.732385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:14.732360 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:14.734260 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:05:14.734228 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6013fca0_3eed_4761_a223_51726201834b.slice/crio-981058602cc957cc6b8f955d127d84c22e85d30901bee8af1ca6b98434d90a9e WatchSource:0}: Error finding container 981058602cc957cc6b8f955d127d84c22e85d30901bee8af1ca6b98434d90a9e: Status 404 returned error can't find the container with id 981058602cc957cc6b8f955d127d84c22e85d30901bee8af1ca6b98434d90a9e Sep 08 17:05:15.344122 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.344092 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" path="/var/lib/kubelet/pods/6a8e901c-5a91-41f9-b2c7-502d4a36db6d/volumes" Sep 08 17:05:15.583005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.582850 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" event={"ID":"a4242b55-9db4-431c-999c-59a91d492a58","Type":"ContainerStarted","Data":"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6"} Sep 08 17:05:15.583005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.582920 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" event={"ID":"a4242b55-9db4-431c-999c-59a91d492a58","Type":"ContainerStarted","Data":"fabf9e79ea06152d9b4a7bea67537ad42d72154044bdc0c97ee3223f049e8338"} Sep 08 17:05:15.583231 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.583018 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:15.584544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.584517 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" event={"ID":"6013fca0-3eed-4761-a223-51726201834b","Type":"ContainerStarted","Data":"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9"} Sep 08 17:05:15.584657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.584566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" event={"ID":"6013fca0-3eed-4761-a223-51726201834b","Type":"ContainerStarted","Data":"981058602cc957cc6b8f955d127d84c22e85d30901bee8af1ca6b98434d90a9e"} Sep 08 17:05:15.584657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.584595 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:15.604019 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.603964 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" podStartSLOduration=1.603948097 podStartE2EDuration="1.603948097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:15.601631546 +0000 UTC m=+846.746402107" watchObservedRunningTime="2026-09-08 17:05:15.603948097 +0000 UTC m=+846.748718657" Sep 08 17:05:15.620194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:15.620147 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" podStartSLOduration=1.620133976 podStartE2EDuration="1.620133976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:15.619006676 +0000 UTC m=+846.763777259" watchObservedRunningTime="2026-09-08 17:05:15.620133976 +0000 UTC m=+846.764904566" Sep 08 17:05:26.592079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:26.592048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:26.592490 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:26.592474 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:29.125355 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:29.125313 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:05:33.691565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.691532 2556 generic.go:356] "Generic (PLEG): container finished" podID="11350a39-3813-4057-926f-de83628927e4" containerID="3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819" exitCode=0 Sep 08 17:05:33.691993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.691588 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerDied","Data":"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819"} Sep 08 17:05:33.691993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.691978 2556 scope.go:117] "RemoveContainer" containerID="3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819" Sep 08 17:05:33.693089 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.693059 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:33.693151 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.693126 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:33.693196 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:33.693157 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:34.698416 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:34.698379 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerStarted","Data":"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c"} Sep 08 17:05:35.181139 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:35.181096 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:35.181400 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:35.181379 2556 patch_prober.go:28] interesting pod/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 08 17:05:35.181474 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:35.181428 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 08 17:05:36.181092 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.181047 2556 patch_prober.go:28] interesting pod/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 08 17:05:36.181612 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.181129 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 08 17:05:36.281233 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.281196 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z"] Sep 08 17:05:36.281681 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.281667 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:36.281681 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.281683 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:36.281783 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.281774 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:36.283764 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.283749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:36.286675 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.286651 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:05:36.286784 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.286651 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:05:36.294033 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.294005 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z"] Sep 08 17:05:36.382917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.382884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z\" (UID: \"323612bf-610e-41cd-bceb-aa9eda030b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:36.484198 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.484099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z\" (UID: \"323612bf-610e-41cd-bceb-aa9eda030b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:36.484828 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.484782 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z\" (UID: \"323612bf-610e-41cd-bceb-aa9eda030b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:36.595302 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.595224 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:36.888185 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:36.888160 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z"] Sep 08 17:05:36.890113 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:05:36.890074 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323612bf_610e_41cd_bceb_aa9eda030b35.slice/crio-7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2 WatchSource:0}: Error finding container 7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2: Status 404 returned error can't find the container with id 7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2 Sep 08 17:05:37.180803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:37.180773 2556 patch_prober.go:28] interesting pod/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 08 17:05:37.180930 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:37.180828 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 08 17:05:37.722190 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:37.722153 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" event={"ID":"323612bf-610e-41cd-bceb-aa9eda030b35","Type":"ContainerStarted","Data":"5e1b900b37e7c3b0676414f223797653a3b3a98bb60f9d1df2c0da33fd1ace17"} Sep 08 17:05:37.722708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:37.722205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" event={"ID":"323612bf-610e-41cd-bceb-aa9eda030b35","Type":"ContainerStarted","Data":"7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2"} Sep 08 17:05:37.739590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:37.739540 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" podStartSLOduration=1.7395236619999999 podStartE2EDuration="1.739523662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:37.739003309 +0000 UTC m=+868.883773871" watchObservedRunningTime="2026-09-08 17:05:37.739523662 +0000 UTC m=+868.884294223" Sep 08 17:05:38.180884 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:38.180849 2556 patch_prober.go:28] interesting pod/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 08 17:05:38.181138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:38.180909 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 08 17:05:38.730061 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:38.730018 2556 generic.go:356] "Generic (PLEG): container finished" podID="323612bf-610e-41cd-bceb-aa9eda030b35" containerID="5e1b900b37e7c3b0676414f223797653a3b3a98bb60f9d1df2c0da33fd1ace17" exitCode=0 Sep 08 17:05:38.730620 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:38.730103 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" event={"ID":"323612bf-610e-41cd-bceb-aa9eda030b35","Type":"ContainerDied","Data":"5e1b900b37e7c3b0676414f223797653a3b3a98bb60f9d1df2c0da33fd1ace17"} Sep 08 17:05:39.180989 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:39.180944 2556 patch_prober.go:28] interesting pod/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 08 17:05:39.181203 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:39.181013 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 08 17:05:39.890894 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:39.890869 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:40.032756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.032655 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca\") pod \"323612bf-610e-41cd-bceb-aa9eda030b35\" (UID: \"323612bf-610e-41cd-bceb-aa9eda030b35\") " Sep 08 17:05:40.033017 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.032991 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "323612bf-610e-41cd-bceb-aa9eda030b35" (UID: "323612bf-610e-41cd-bceb-aa9eda030b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:40.133445 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.133414 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323612bf-610e-41cd-bceb-aa9eda030b35-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:40.185649 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.185613 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:40.185917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.185890 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:40.186593 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.186575 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:40.744871 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.744830 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" event={"ID":"323612bf-610e-41cd-bceb-aa9eda030b35","Type":"ContainerDied","Data":"7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2"} Sep 08 17:05:40.744871 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.744857 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z" Sep 08 17:05:40.745086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:40.744894 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec6782b30c3537010dec38ae325112109727c45f8ef5035c6efd287b1173ec2" Sep 08 17:05:44.471602 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.471565 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:44.471996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.471925 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" podUID="a4242b55-9db4-431c-999c-59a91d492a58" containerName="payload-processing" containerID="cri-o://a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6" gracePeriod=30 Sep 08 17:05:44.481187 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.481159 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:44.481419 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.481368 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" podUID="6013fca0-3eed-4761-a223-51726201834b" containerName="payload-pre-processing" containerID="cri-o://19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9" gracePeriod=30 Sep 08 17:05:44.746517 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.746492 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:44.757881 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.757860 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:44.769716 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.769668 2556 generic.go:356] "Generic (PLEG): container finished" podID="a4242b55-9db4-431c-999c-59a91d492a58" containerID="a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6" exitCode=0 Sep 08 17:05:44.769716 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.769709 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" Sep 08 17:05:44.769926 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.769813 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" event={"ID":"a4242b55-9db4-431c-999c-59a91d492a58","Type":"ContainerDied","Data":"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6"} Sep 08 17:05:44.770209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.770168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56" event={"ID":"a4242b55-9db4-431c-999c-59a91d492a58","Type":"ContainerDied","Data":"fabf9e79ea06152d9b4a7bea67537ad42d72154044bdc0c97ee3223f049e8338"} Sep 08 17:05:44.770304 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.770223 2556 scope.go:117] "RemoveContainer" containerID="a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6" Sep 08 17:05:44.772517 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.772490 2556 generic.go:356] "Generic (PLEG): container finished" podID="6013fca0-3eed-4761-a223-51726201834b" containerID="19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9" exitCode=0 Sep 08 17:05:44.772645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.772533 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" Sep 08 17:05:44.772741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.772533 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" event={"ID":"6013fca0-3eed-4761-a223-51726201834b","Type":"ContainerDied","Data":"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9"} Sep 08 17:05:44.772817 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.772733 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6" event={"ID":"6013fca0-3eed-4761-a223-51726201834b","Type":"ContainerDied","Data":"981058602cc957cc6b8f955d127d84c22e85d30901bee8af1ca6b98434d90a9e"} Sep 08 17:05:44.782764 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.782745 2556 scope.go:117] "RemoveContainer" containerID="a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6" Sep 08 17:05:44.783890 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:44.783869 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6\": container with ID starting with a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6 not found: ID does not exist" containerID="a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6" Sep 08 17:05:44.783989 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.783898 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6"} err="failed to get container status \"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6\": rpc error: code = NotFound desc = could not find container \"a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6\": container with ID starting with a7427e627e8344daf45b21038f23b10f27bc140adcd98c5303e3ce62298ceba6 not found: ID does not exist" Sep 08 17:05:44.783989 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.783915 2556 scope.go:117] "RemoveContainer" containerID="19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9" Sep 08 17:05:44.797892 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.797865 2556 scope.go:117] "RemoveContainer" containerID="19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9" Sep 08 17:05:44.798282 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:44.798259 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9\": container with ID starting with 19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9 not found: ID does not exist" containerID="19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9" Sep 08 17:05:44.798394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.798290 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9"} err="failed to get container status \"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9\": rpc error: code = NotFound desc = could not find container \"19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9\": container with ID starting with 19b8d0c67e86fb230880855a31c86d937fe0b2e27029da80c4f35787be0826b9 not found: ID does not exist" Sep 08 17:05:44.800295 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.800263 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:44.800530 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.800509 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" containerID="cri-o://7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c" gracePeriod=30 Sep 08 17:05:44.887552 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887515 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume\") pod \"6013fca0-3eed-4761-a223-51726201834b\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " Sep 08 17:05:44.887740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887573 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume\") pod \"a4242b55-9db4-431c-999c-59a91d492a58\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " Sep 08 17:05:44.887740 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887651 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ccjd\" (UniqueName: \"kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd\") pod \"6013fca0-3eed-4761-a223-51726201834b\" (UID: \"6013fca0-3eed-4761-a223-51726201834b\") " Sep 08 17:05:44.887865 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887737 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz55t\" (UniqueName: \"kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t\") pod \"a4242b55-9db4-431c-999c-59a91d492a58\" (UID: \"a4242b55-9db4-431c-999c-59a91d492a58\") " Sep 08 17:05:44.887917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887881 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6013fca0-3eed-4761-a223-51726201834b" (UID: "6013fca0-3eed-4761-a223-51726201834b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:44.887917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.887882 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4242b55-9db4-431c-999c-59a91d492a58" (UID: "a4242b55-9db4-431c-999c-59a91d492a58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:44.888139 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.888122 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013fca0-3eed-4761-a223-51726201834b-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.888198 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.888144 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4242b55-9db4-431c-999c-59a91d492a58-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.889910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.889877 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t" (OuterVolumeSpecName: "kube-api-access-vz55t") pod "a4242b55-9db4-431c-999c-59a91d492a58" (UID: "a4242b55-9db4-431c-999c-59a91d492a58"). InnerVolumeSpecName "kube-api-access-vz55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:44.889986 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.889915 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd" (OuterVolumeSpecName: "kube-api-access-2ccjd") pod "6013fca0-3eed-4761-a223-51726201834b" (UID: "6013fca0-3eed-4761-a223-51726201834b"). InnerVolumeSpecName "kube-api-access-2ccjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:44.989272 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.989184 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz55t\" (UniqueName: \"kubernetes.io/projected/a4242b55-9db4-431c-999c-59a91d492a58-kube-api-access-vz55t\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:44.989272 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:44.989212 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccjd\" (UniqueName: \"kubernetes.io/projected/6013fca0-3eed-4761-a223-51726201834b-kube-api-access-2ccjd\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:45.094756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.094723 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:45.101015 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.100989 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b709328f-589c9f566f-ktt56"] Sep 08 17:05:45.113177 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.113145 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:45.119825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.119803 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b709328f-6bf65b94db-zwbt6"] Sep 08 17:05:45.344245 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.344168 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6013fca0-3eed-4761-a223-51726201834b" path="/var/lib/kubelet/pods/6013fca0-3eed-4761-a223-51726201834b/volumes" Sep 08 17:05:45.344604 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:45.344590 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4242b55-9db4-431c-999c-59a91d492a58" path="/var/lib/kubelet/pods/a4242b55-9db4-431c-999c-59a91d492a58/volumes" Sep 08 17:05:47.442631 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:47.442597 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z"] Sep 08 17:05:47.446532 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:47.446510 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-66e7a8-9vq2z"] Sep 08 17:05:49.343471 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:49.343437 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323612bf-610e-41cd-bceb-aa9eda030b35" path="/var/lib/kubelet/pods/323612bf-610e-41cd-bceb-aa9eda030b35/volumes" Sep 08 17:05:49.527173 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:49.527142 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:05:50.045543 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.045520 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:50.147691 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147647 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147714 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qwvv\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147750 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147767 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147796 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147814 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147833 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.147875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147860 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.148253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.147904 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs\") pod \"11350a39-3813-4057-926f-de83628927e4\" (UID: \"11350a39-3813-4057-926f-de83628927e4\") " Sep 08 17:05:50.148253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148036 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:05:50.148394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148272 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:50.148394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148373 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:50.148515 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148391 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:50.148515 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148503 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:50.148608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148522 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11350a39-3813-4057-926f-de83628927e4-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.148608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148541 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.148608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148555 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.148608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.148569 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.149951 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.149929 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:50.149951 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.149941 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:05:50.150213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.150198 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv" (OuterVolumeSpecName: "kube-api-access-9qwvv") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "kube-api-access-9qwvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:05:50.150292 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.150279 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11350a39-3813-4057-926f-de83628927e4" (UID: "11350a39-3813-4057-926f-de83628927e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:05:50.250088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.250045 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwvv\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-kube-api-access-9qwvv\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.250088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.250087 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.250306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.250101 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11350a39-3813-4057-926f-de83628927e4-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.250306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.250114 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11350a39-3813-4057-926f-de83628927e4-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.250306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.250127 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11350a39-3813-4057-926f-de83628927e4-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:05:50.809468 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.809430 2556 generic.go:356] "Generic (PLEG): container finished" podID="11350a39-3813-4057-926f-de83628927e4" containerID="7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c" exitCode=0 Sep 08 17:05:50.809896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.809477 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" Sep 08 17:05:50.809896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.809511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerDied","Data":"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c"} Sep 08 17:05:50.809896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.809564 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r" event={"ID":"11350a39-3813-4057-926f-de83628927e4","Type":"ContainerDied","Data":"faafac4b15a9403485c2ca7997d50355e68db6c72f52efc337245f4067ef1374"} Sep 08 17:05:50.809896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.809586 2556 scope.go:117] "RemoveContainer" containerID="7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c" Sep 08 17:05:50.819002 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.818987 2556 scope.go:117] "RemoveContainer" containerID="3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819" Sep 08 17:05:50.826773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.826760 2556 scope.go:117] "RemoveContainer" containerID="7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c" Sep 08 17:05:50.827029 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:50.827010 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c\": container with ID starting with 7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c not found: ID does not exist" containerID="7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c" Sep 08 17:05:50.827100 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.827040 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c"} err="failed to get container status \"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c\": rpc error: code = NotFound desc = could not find container \"7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c\": container with ID starting with 7184762d7e8862e81a09e208307f7308c2521fb552ee26546584ddefb745488c not found: ID does not exist" Sep 08 17:05:50.827100 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.827063 2556 scope.go:117] "RemoveContainer" containerID="3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819" Sep 08 17:05:50.827346 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:05:50.827314 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819\": container with ID starting with 3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819 not found: ID does not exist" containerID="3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819" Sep 08 17:05:50.827385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.827355 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819"} err="failed to get container status \"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819\": rpc error: code = NotFound desc = could not find container \"3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819\": container with ID starting with 3ff1742a4765e64d29df4e90f8b86bfd4b34dd5439510b8a4d90f1721390b819 not found: ID does not exist" Sep 08 17:05:50.833144 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.833122 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:50.843651 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:50.843627 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b709328f-openshift-default-86dcf7859b-qtw7r"] Sep 08 17:05:51.342665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:51.342631 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11350a39-3813-4057-926f-de83628927e4" path="/var/lib/kubelet/pods/11350a39-3813-4057-926f-de83628927e4/volumes" Sep 08 17:05:55.003693 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.003658 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:05:55.004050 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.003917 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" containerID="cri-o://2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658" gracePeriod=30 Sep 08 17:05:55.749692 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.749658 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:05:55.750108 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750093 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4242b55-9db4-431c-999c-59a91d492a58" containerName="payload-processing" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750110 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4242b55-9db4-431c-999c-59a91d492a58" containerName="payload-processing" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750122 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750128 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750138 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6013fca0-3eed-4761-a223-51726201834b" containerName="payload-pre-processing" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750144 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6013fca0-3eed-4761-a223-51726201834b" containerName="payload-pre-processing" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750150 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323612bf-610e-41cd-bceb-aa9eda030b35" containerName="revoke-keys" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750155 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="323612bf-610e-41cd-bceb-aa9eda030b35" containerName="revoke-keys" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750162 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:05:55.750174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750167 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:05:55.750477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750265 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:05:55.750477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750276 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4242b55-9db4-431c-999c-59a91d492a58" containerName="payload-processing" Sep 08 17:05:55.750477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750283 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6013fca0-3eed-4761-a223-51726201834b" containerName="payload-pre-processing" Sep 08 17:05:55.750477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750290 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8e901c-5a91-41f9-b2c7-502d4a36db6d" containerName="istio-proxy" Sep 08 17:05:55.750477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.750296 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="323612bf-610e-41cd-bceb-aa9eda030b35" containerName="revoke-keys" Sep 08 17:05:55.753250 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.753233 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.756293 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.756273 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-3b5d3c-openshift-default-dockercfg-lkx4r\"" Sep 08 17:05:55.765982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.765960 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:05:55.806560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806533 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806575 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806597 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vx4\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806674 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806884 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806884 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.806884 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.806791 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907547 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907547 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907550 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907570 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vx4\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907644 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.907803 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907754 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908094 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907848 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908094 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.907985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.908194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908306 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.908246 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.908371 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.908486 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.908462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.910141 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.910121 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.910268 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.910249 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.915596 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.915573 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:55.915708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:55.915598 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vx4\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4\") pod \"e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:56.064138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.064059 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:56.342737 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.342535 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:05:56.345375 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:05:56.345347 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7f0e9c_d77f_4698_9ec3_c1b88d25f867.slice/crio-582a08d9fe3737e678fefecc0ca8d6902843bc16c6a0dd33d5fd57769bc575bf WatchSource:0}: Error finding container 582a08d9fe3737e678fefecc0ca8d6902843bc16c6a0dd33d5fd57769bc575bf: Status 404 returned error can't find the container with id 582a08d9fe3737e678fefecc0ca8d6902843bc16c6a0dd33d5fd57769bc575bf Sep 08 17:05:56.347474 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.347444 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:56.347549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.347506 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:56.347549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.347535 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:05:56.844535 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.844499 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerStarted","Data":"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752"} Sep 08 17:05:56.844747 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.844557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerStarted","Data":"582a08d9fe3737e678fefecc0ca8d6902843bc16c6a0dd33d5fd57769bc575bf"} Sep 08 17:05:56.868931 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:56.868876 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podStartSLOduration=1.868861018 podStartE2EDuration="1.868861018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:56.866215238 +0000 UTC m=+888.010985813" watchObservedRunningTime="2026-09-08 17:05:56.868861018 +0000 UTC m=+888.013631578" Sep 08 17:05:57.064982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:57.064942 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:57.066283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:57.066258 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:05:57.066364 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:57.066336 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:05:58.068480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:58.068446 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:58.856413 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:58.856379 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:05:58.857594 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:05:58.857569 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:06:00.254981 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.254956 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:06:00.352261 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352179 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352261 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352219 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352268 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352294 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352349 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352374 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r2v5\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352399 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352420 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352497 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy\") pod \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\" (UID: \"1ace42ef-e47e-46de-9e0c-7c75c486ccd4\") " Sep 08 17:06:00.352872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352499 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:06:00.352872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352632 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:06:00.352872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352646 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:06:00.352872 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352709 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:06:00.353079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352919 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.353079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352933 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.353079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352949 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.353079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352963 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.353079 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.352923 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:06:00.354581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.354555 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:06:00.354692 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.354595 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:06:00.354692 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.354593 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:06:00.354821 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.354806 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5" (OuterVolumeSpecName: "kube-api-access-4r2v5") pod "1ace42ef-e47e-46de-9e0c-7c75c486ccd4" (UID: "1ace42ef-e47e-46de-9e0c-7c75c486ccd4"). InnerVolumeSpecName "kube-api-access-4r2v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:06:00.453733 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.453697 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.453733 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.453725 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.453733 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.453733 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.453953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.453742 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2v5\" (UniqueName: \"kubernetes.io/projected/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-kube-api-access-4r2v5\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.453953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.453752 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ace42ef-e47e-46de-9e0c-7c75c486ccd4-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:00.868669 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.868635 2556 generic.go:356] "Generic (PLEG): container finished" podID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerID="2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658" exitCode=0 Sep 08 17:06:00.868854 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.868683 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" Sep 08 17:06:00.868854 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.868724 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerDied","Data":"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658"} Sep 08 17:06:00.868854 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.868785 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb" event={"ID":"1ace42ef-e47e-46de-9e0c-7c75c486ccd4","Type":"ContainerDied","Data":"4491ec57da0207bc38e0cfdfa5419c36c8727c924c472a74a392f647a882298c"} Sep 08 17:06:00.868854 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.868813 2556 scope.go:117] "RemoveContainer" containerID="2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658" Sep 08 17:06:00.878192 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.878175 2556 scope.go:117] "RemoveContainer" containerID="2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777" Sep 08 17:06:00.886646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.886629 2556 scope.go:117] "RemoveContainer" containerID="2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658" Sep 08 17:06:00.886907 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:06:00.886890 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658\": container with ID starting with 2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658 not found: ID does not exist" containerID="2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658" Sep 08 17:06:00.886960 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.886916 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658"} err="failed to get container status \"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658\": rpc error: code = NotFound desc = could not find container \"2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658\": container with ID starting with 2506c8eac09e6f6dafaa99c4e4ca639ffbfd6efd05ae4659911e487d22a24658 not found: ID does not exist" Sep 08 17:06:00.886960 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.886932 2556 scope.go:117] "RemoveContainer" containerID="2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777" Sep 08 17:06:00.887142 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:06:00.887127 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777\": container with ID starting with 2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777 not found: ID does not exist" containerID="2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777" Sep 08 17:06:00.887179 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.887146 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777"} err="failed to get container status \"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777\": rpc error: code = NotFound desc = could not find container \"2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777\": container with ID starting with 2796ff5e1028a4d521daaca9093bc3b126ec9a397b0a507a1700e5261b6c7777 not found: ID does not exist" Sep 08 17:06:00.898774 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.898754 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:06:00.910130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.910107 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-66e7a8-openshift-default-8d7bbf6cb-h5vsb"] Sep 08 17:06:00.939020 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.938994 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:00.939407 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939395 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.939451 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939410 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.939451 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939423 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.939451 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939433 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.939549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939455 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:06:00.939549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939461 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:06:00.939549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939521 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11350a39-3813-4057-926f-de83628927e4" containerName="istio-proxy" Sep 08 17:06:00.939549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939529 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.939549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.939536 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" containerName="istio-proxy" Sep 08 17:06:00.941982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.941966 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:00.945197 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.945177 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3b5d3c\"" Sep 08 17:06:00.945299 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.945232 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3b5d3c-dockercfg-k5pcj\"" Sep 08 17:06:00.958839 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:00.958818 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:01.059825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.059786 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcjmr\" (UniqueName: \"kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.059994 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.059972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.161269 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.161187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.161269 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.161239 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcjmr\" (UniqueName: \"kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.161933 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.161879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.178958 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.178932 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcjmr\" (UniqueName: \"kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr\") pod \"payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.252024 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.251993 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.343617 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.343585 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ace42ef-e47e-46de-9e0c-7c75c486ccd4" path="/var/lib/kubelet/pods/1ace42ef-e47e-46de-9e0c-7c75c486ccd4/volumes" Sep 08 17:06:01.535063 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.534944 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:01.537858 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:06:01.537832 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ad20c8_0933_4ed5_b0db_688eed5c4197.slice/crio-c260c3a50d9ed7acd739212dde924f7bd539d3a320ecd8af02afc3410c84b721 WatchSource:0}: Error finding container c260c3a50d9ed7acd739212dde924f7bd539d3a320ecd8af02afc3410c84b721: Status 404 returned error can't find the container with id c260c3a50d9ed7acd739212dde924f7bd539d3a320ecd8af02afc3410c84b721 Sep 08 17:06:01.875591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.875563 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" event={"ID":"33ad20c8-0933-4ed5-b0db-688eed5c4197","Type":"ContainerStarted","Data":"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01"} Sep 08 17:06:01.875763 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.875697 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:01.875763 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.875748 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" event={"ID":"33ad20c8-0933-4ed5-b0db-688eed5c4197","Type":"ContainerStarted","Data":"c260c3a50d9ed7acd739212dde924f7bd539d3a320ecd8af02afc3410c84b721"} Sep 08 17:06:01.909612 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:01.909563 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" podStartSLOduration=1.9095478689999998 podStartE2EDuration="1.909547869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:01.907618991 +0000 UTC m=+893.052389552" watchObservedRunningTime="2026-09-08 17:06:01.909547869 +0000 UTC m=+893.054318431" Sep 08 17:06:05.346873 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:05.346844 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:06:12.885523 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:12.885487 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:14.898515 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:14.898462 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 17:06:14.898998 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:14.898546 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 17:06:15.956716 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.956676 2556 generic.go:356] "Generic (PLEG): container finished" podID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerID="cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752" exitCode=0 Sep 08 17:06:15.957279 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.956743 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerDied","Data":"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752"} Sep 08 17:06:15.957279 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.957250 2556 scope.go:117] "RemoveContainer" containerID="cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752" Sep 08 17:06:15.958677 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.958641 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:06:15.958787 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.958733 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:06:15.958787 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:15.958781 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:06:16.963855 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:16.963824 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerStarted","Data":"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88"} Sep 08 17:06:17.065334 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:17.065279 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:06:17.065521 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:17.065499 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:17.065580 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:17.065546 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:18.064992 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:18.064954 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:18.065582 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:18.065551 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:19.065579 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:19.065535 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:19.066070 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:19.065603 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:20.064652 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:20.064600 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:20.064880 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:20.064675 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:21.065375 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:21.065312 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:21.065874 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:21.065435 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:22.065173 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:22.065125 2556 patch_prober.go:28] interesting pod/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 08 17:06:22.065421 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:22.065210 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 08 17:06:22.617539 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:22.617499 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:06:23.068739 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:23.068651 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:06:23.069152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:23.069088 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:06:23.069965 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:23.069941 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:06:37.704982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:37.704944 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:06:47.588119 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.588067 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7"] Sep 08 17:06:47.591908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.591879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:47.594759 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.594726 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:06:47.594893 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.594726 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:06:47.600528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.600467 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7"] Sep 08 17:06:47.677173 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.677133 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7\" (UID: \"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:47.777927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.777894 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7\" (UID: \"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:47.778537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.778516 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7\" (UID: \"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:47.907789 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:47.907759 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:48.178767 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:48.178736 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7"] Sep 08 17:06:48.181706 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:06:48.181672 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb0a3a1_74cd_4148_b0d1_6b823cf3203a.slice/crio-793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237 WatchSource:0}: Error finding container 793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237: Status 404 returned error can't find the container with id 793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237 Sep 08 17:06:49.124214 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:49.124180 2556 generic.go:356] "Generic (PLEG): container finished" podID="8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" containerID="2171306cc5f290c647015d58c6fcbc322290f5bbc577f30828f3a93a2d0dca71" exitCode=0 Sep 08 17:06:49.124603 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:49.124252 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" event={"ID":"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a","Type":"ContainerDied","Data":"2171306cc5f290c647015d58c6fcbc322290f5bbc577f30828f3a93a2d0dca71"} Sep 08 17:06:49.124603 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:49.124297 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" event={"ID":"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a","Type":"ContainerStarted","Data":"793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237"} Sep 08 17:06:50.256960 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:50.256934 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:50.305333 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:50.305303 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca\") pod \"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a\" (UID: \"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a\") " Sep 08 17:06:50.305645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:50.305622 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" (UID: "8eb0a3a1-74cd-4148-b0d1-6b823cf3203a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:06:50.406584 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:50.406498 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:51.135048 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:51.135019 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" Sep 08 17:06:51.135248 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:51.135066 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7" event={"ID":"8eb0a3a1-74cd-4148-b0d1-6b823cf3203a","Type":"ContainerDied","Data":"793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237"} Sep 08 17:06:51.135248 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:51.135117 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793f6fdd6edbb10bc03fc00fca7cbd50d845ffa3ef4dad911aef5ea7d6e95237" Sep 08 17:06:54.805115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:54.805082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:06:58.235036 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.234998 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:58.235517 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.235289 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" podUID="33ad20c8-0933-4ed5-b0db-688eed5c4197" containerName="payload-pre-processing" containerID="cri-o://1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01" gracePeriod=30 Sep 08 17:06:58.443317 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.443282 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7"] Sep 08 17:06:58.449728 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.449689 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3b5d3c-55pn7"] Sep 08 17:06:58.477630 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.477609 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:58.593467 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.593371 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume\") pod \"33ad20c8-0933-4ed5-b0db-688eed5c4197\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " Sep 08 17:06:58.593467 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.593409 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcjmr\" (UniqueName: \"kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr\") pod \"33ad20c8-0933-4ed5-b0db-688eed5c4197\" (UID: \"33ad20c8-0933-4ed5-b0db-688eed5c4197\") " Sep 08 17:06:58.593754 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.593724 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33ad20c8-0933-4ed5-b0db-688eed5c4197" (UID: "33ad20c8-0933-4ed5-b0db-688eed5c4197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:06:58.595449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.595428 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr" (OuterVolumeSpecName: "kube-api-access-vcjmr") pod "33ad20c8-0933-4ed5-b0db-688eed5c4197" (UID: "33ad20c8-0933-4ed5-b0db-688eed5c4197"). InnerVolumeSpecName "kube-api-access-vcjmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:06:58.694581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.694547 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ad20c8-0933-4ed5-b0db-688eed5c4197-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:58.694581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:58.694575 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcjmr\" (UniqueName: \"kubernetes.io/projected/33ad20c8-0933-4ed5-b0db-688eed5c4197-kube-api-access-vcjmr\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:06:59.179561 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.179527 2556 generic.go:356] "Generic (PLEG): container finished" podID="33ad20c8-0933-4ed5-b0db-688eed5c4197" containerID="1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01" exitCode=0 Sep 08 17:06:59.179711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.179574 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" Sep 08 17:06:59.179711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.179601 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" event={"ID":"33ad20c8-0933-4ed5-b0db-688eed5c4197","Type":"ContainerDied","Data":"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01"} Sep 08 17:06:59.179711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.179641 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd" event={"ID":"33ad20c8-0933-4ed5-b0db-688eed5c4197","Type":"ContainerDied","Data":"c260c3a50d9ed7acd739212dde924f7bd539d3a320ecd8af02afc3410c84b721"} Sep 08 17:06:59.179711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.179663 2556 scope.go:117] "RemoveContainer" containerID="1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01" Sep 08 17:06:59.188650 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.188628 2556 scope.go:117] "RemoveContainer" containerID="1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01" Sep 08 17:06:59.188914 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:06:59.188896 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01\": container with ID starting with 1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01 not found: ID does not exist" containerID="1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01" Sep 08 17:06:59.188967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.188921 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01"} err="failed to get container status \"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01\": rpc error: code = NotFound desc = could not find container \"1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01\": container with ID starting with 1f898bf332c4ca2e4369dca5568f6e428ed8276943958fffd139061fc32bea01 not found: ID does not exist" Sep 08 17:06:59.201356 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.201313 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:59.207225 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.207202 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3b5d3c-7756b4859d-gsfhd"] Sep 08 17:06:59.344269 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.344238 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ad20c8-0933-4ed5-b0db-688eed5c4197" path="/var/lib/kubelet/pods/33ad20c8-0933-4ed5-b0db-688eed5c4197/volumes" Sep 08 17:06:59.344661 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:06:59.344616 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" path="/var/lib/kubelet/pods/8eb0a3a1-74cd-4148-b0d1-6b823cf3203a/volumes" Sep 08 17:07:09.998748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:09.998715 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:07:09.999209 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:09.999094 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" containerID="cri-o://59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88" gracePeriod=30 Sep 08 17:07:10.035073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.035033 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:07:10.739882 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.739845 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:10.740284 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740272 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" containerName="revoke-keys" Sep 08 17:07:10.740349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740285 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" containerName="revoke-keys" Sep 08 17:07:10.740349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740301 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ad20c8-0933-4ed5-b0db-688eed5c4197" containerName="payload-pre-processing" Sep 08 17:07:10.740349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740306 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ad20c8-0933-4ed5-b0db-688eed5c4197" containerName="payload-pre-processing" Sep 08 17:07:10.740449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740378 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ad20c8-0933-4ed5-b0db-688eed5c4197" containerName="payload-pre-processing" Sep 08 17:07:10.740449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.740387 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb0a3a1-74cd-4148-b0d1-6b823cf3203a" containerName="revoke-keys" Sep 08 17:07:10.742790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.742771 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.745589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.745570 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-79e919-openshift-default-dockercfg-x4ssb\"" Sep 08 17:07:10.756088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.756062 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:10.917143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917102 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917313 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917152 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917313 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917237 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917313 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917446 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917446 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917397 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917518 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917552 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917523 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqt8\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:10.917552 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:10.917546 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.018504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018407 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.018504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018479 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqt8\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018592 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018860 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018927 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018977 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.018989 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019392 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.019055 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019392 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.019102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019508 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.019484 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.019666 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.019645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.020921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.020898 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.021088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.021070 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.027224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.027200 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.027312 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.027258 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqt8\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8\") pod \"e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.053311 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.053285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:11.333274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.333243 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:11.334982 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:07:11.334945 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054bd77b_1916_4888_a7fc_b4cc21658063.slice/crio-1420fd451887ebd9a2913f20ca3b59b77b744d1a915e71768222e672e4cae8ec WatchSource:0}: Error finding container 1420fd451887ebd9a2913f20ca3b59b77b744d1a915e71768222e672e4cae8ec: Status 404 returned error can't find the container with id 1420fd451887ebd9a2913f20ca3b59b77b744d1a915e71768222e672e4cae8ec Sep 08 17:07:11.343363 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.338638 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:11.343363 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.339063 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:11.343363 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:11.339530 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:12.251252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:12.251221 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" event={"ID":"054bd77b-1916-4888-a7fc-b4cc21658063","Type":"ContainerStarted","Data":"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f"} Sep 08 17:07:12.251641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:12.251273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" event={"ID":"054bd77b-1916-4888-a7fc-b4cc21658063","Type":"ContainerStarted","Data":"1420fd451887ebd9a2913f20ca3b59b77b744d1a915e71768222e672e4cae8ec"} Sep 08 17:07:12.275921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:12.275868 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" podStartSLOduration=2.275852737 podStartE2EDuration="2.275852737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:12.27388432 +0000 UTC m=+963.418654883" watchObservedRunningTime="2026-09-08 17:07:12.275852737 +0000 UTC m=+963.420623375" Sep 08 17:07:13.053970 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:13.053931 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:13.059002 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:13.058972 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:13.257444 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:13.257408 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:13.258253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:13.258232 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:15.252616 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.252591 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:07:15.258854 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.258837 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.258921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.258869 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.258958 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.258924 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.258958 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.258947 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.258972 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259001 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259041 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259115 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4vx4\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259154 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert\") pod \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\" (UID: \"db7f0e9c-d77f-4698-9ec3-c1b88d25f867\") " Sep 08 17:07:15.259242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259203 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:15.259375 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259254 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data" (OuterVolumeSpecName: "istio-data") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:15.259375 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259275 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:15.259504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259465 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:15.259663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259613 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:07:15.259663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259642 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.259663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259661 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.259842 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259675 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.259842 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.259688 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.261105 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.261081 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:15.261105 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.261096 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:07:15.261236 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.261120 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token" (OuterVolumeSpecName: "istio-token") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:07:15.261288 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.261275 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4" (OuterVolumeSpecName: "kube-api-access-d4vx4") pod "db7f0e9c-d77f-4698-9ec3-c1b88d25f867" (UID: "db7f0e9c-d77f-4698-9ec3-c1b88d25f867"). InnerVolumeSpecName "kube-api-access-d4vx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:07:15.272104 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.272080 2556 generic.go:356] "Generic (PLEG): container finished" podID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerID="59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88" exitCode=0 Sep 08 17:07:15.272225 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.272128 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" Sep 08 17:07:15.272225 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.272164 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerDied","Data":"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88"} Sep 08 17:07:15.272368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.272229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n" event={"ID":"db7f0e9c-d77f-4698-9ec3-c1b88d25f867","Type":"ContainerDied","Data":"582a08d9fe3737e678fefecc0ca8d6902843bc16c6a0dd33d5fd57769bc575bf"} Sep 08 17:07:15.272368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.272265 2556 scope.go:117] "RemoveContainer" containerID="59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88" Sep 08 17:07:15.281000 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.280981 2556 scope.go:117] "RemoveContainer" containerID="cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752" Sep 08 17:07:15.288472 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.288456 2556 scope.go:117] "RemoveContainer" containerID="59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88" Sep 08 17:07:15.288722 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:07:15.288704 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88\": container with ID starting with 59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88 not found: ID does not exist" containerID="59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88" Sep 08 17:07:15.288801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.288734 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88"} err="failed to get container status \"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88\": rpc error: code = NotFound desc = could not find container \"59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88\": container with ID starting with 59e6642953734b956bea9c018eb821f986e0744926dfa091dc6ef34f83ebff88 not found: ID does not exist" Sep 08 17:07:15.288801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.288757 2556 scope.go:117] "RemoveContainer" containerID="cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752" Sep 08 17:07:15.289024 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:07:15.289008 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752\": container with ID starting with cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752 not found: ID does not exist" containerID="cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752" Sep 08 17:07:15.289065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.289029 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752"} err="failed to get container status \"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752\": rpc error: code = NotFound desc = could not find container \"cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752\": container with ID starting with cf90ac4b91cbe426a860dc4bc7142f3bec2a5657c327088fa935926248787752 not found: ID does not exist" Sep 08 17:07:15.327660 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.327629 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:07:15.352820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.352790 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-3b5d3c-openshift-default-6cc8964b7b-nt52n"] Sep 08 17:07:15.361126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.361098 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vx4\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-kube-api-access-d4vx4\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.361283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.361129 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.361283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.361146 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.361283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.361159 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:15.361283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:15.361172 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f0e9c-d77f-4698-9ec3-c1b88d25f867-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:17.343857 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:17.343819 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" path="/var/lib/kubelet/pods/db7f0e9c-d77f-4698-9ec3-c1b88d25f867/volumes" Sep 08 17:07:22.202913 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.202864 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:07:22.203385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.203309 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:22.203385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.203338 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:22.203482 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.203419 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:22.206668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.206651 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.209579 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.209555 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a837a8a9-openshift-default-dockercfg-694zp\"" Sep 08 17:07:22.219423 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.219395 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:07:22.230128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230105 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230245 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230141 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230245 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230193 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230245 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230238 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230372 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230257 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230372 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230372 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230362 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjctm\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230384 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.230473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.230403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.330934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.330895 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.330934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.330936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.330978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331009 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331028 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331133 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjctm\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331190 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331354 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331421 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331506 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.331871 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.331845 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.332263 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.332238 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.333615 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.333593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.334126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.334106 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.339591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.339569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.340378 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.340358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjctm\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm\") pod \"e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.517226 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.517143 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:22.797242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.797159 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:07:22.799606 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:07:22.799569 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f75ebf7_46aa_46a1_9443_a9994194e78b.slice/crio-f315f2af332ac865e9ab1fe721177ffaa87b881772242acfefabe3d32ddf2367 WatchSource:0}: Error finding container f315f2af332ac865e9ab1fe721177ffaa87b881772242acfefabe3d32ddf2367: Status 404 returned error can't find the container with id f315f2af332ac865e9ab1fe721177ffaa87b881772242acfefabe3d32ddf2367 Sep 08 17:07:22.801868 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.801835 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:22.801963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.801898 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:22.801963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:22.801927 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:23.319308 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:23.319266 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerStarted","Data":"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a"} Sep 08 17:07:23.319977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:23.319396 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerStarted","Data":"f315f2af332ac865e9ab1fe721177ffaa87b881772242acfefabe3d32ddf2367"} Sep 08 17:07:23.343675 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:23.343622 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podStartSLOduration=1.343607015 podStartE2EDuration="1.343607015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:23.343055164 +0000 UTC m=+974.487825752" watchObservedRunningTime="2026-09-08 17:07:23.343607015 +0000 UTC m=+974.488377576" Sep 08 17:07:23.517531 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:23.517498 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:23.522817 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:23.522791 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:24.324593 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:24.324562 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:24.325571 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:24.325551 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:26.216733 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:26.216693 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:07:27.307722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.307690 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:07:27.308135 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.308126 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:27.308189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.308139 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:27.308230 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.308218 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7f0e9c-d77f-4698-9ec3-c1b88d25f867" containerName="istio-proxy" Sep 08 17:07:27.310301 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.310286 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.312948 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.312925 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a837a8a9-dockercfg-42h4k\"" Sep 08 17:07:27.313377 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.313364 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a837a8a9\"" Sep 08 17:07:27.322623 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.322602 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:07:27.384584 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.384549 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.384778 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.384608 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6ff\" (UniqueName: \"kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.485770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.485732 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6ff\" (UniqueName: \"kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.485946 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.485863 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.486530 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.486512 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.494561 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.494539 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6ff\" (UniqueName: \"kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff\") pod \"payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.621539 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.621508 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:27.894140 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:27.894113 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:07:27.896315 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:07:27.896286 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f93ee78_30ec_4c1a_b1e4_e4b13f40caf2.slice/crio-8c62fa5a53a6b4f1e5b2795c0dcb04089c360650a020e85e02933561a4892ce2 WatchSource:0}: Error finding container 8c62fa5a53a6b4f1e5b2795c0dcb04089c360650a020e85e02933561a4892ce2: Status 404 returned error can't find the container with id 8c62fa5a53a6b4f1e5b2795c0dcb04089c360650a020e85e02933561a4892ce2 Sep 08 17:07:28.346126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:28.346039 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" event={"ID":"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2","Type":"ContainerStarted","Data":"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f"} Sep 08 17:07:28.346126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:28.346088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" event={"ID":"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2","Type":"ContainerStarted","Data":"8c62fa5a53a6b4f1e5b2795c0dcb04089c360650a020e85e02933561a4892ce2"} Sep 08 17:07:28.346563 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:28.346244 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:28.369555 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:28.369505 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" podStartSLOduration=1.3694892730000001 podStartE2EDuration="1.369489273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:28.368661552 +0000 UTC m=+979.513432114" watchObservedRunningTime="2026-09-08 17:07:28.369489273 +0000 UTC m=+979.514259833" Sep 08 17:07:38.997271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:38.997191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:07:39.353793 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:39.353703 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:07:43.433704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.433619 2556 generic.go:356] "Generic (PLEG): container finished" podID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerID="375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a" exitCode=0 Sep 08 17:07:43.434091 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.433697 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerDied","Data":"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a"} Sep 08 17:07:43.434153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.434114 2556 scope.go:117] "RemoveContainer" containerID="375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a" Sep 08 17:07:43.435196 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.435168 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:43.435334 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.435250 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:43.435334 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:43.435301 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:07:44.441675 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:44.441637 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerStarted","Data":"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6"} Sep 08 17:07:44.517904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:44.517866 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:44.518105 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:44.518081 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:44.518151 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:44.518133 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:45.518103 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:45.518064 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:45.518596 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:45.518126 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:46.517995 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:46.517960 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:46.518202 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:46.518045 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:47.518426 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:47.518388 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:47.518911 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:47.518465 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:48.518194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.518157 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:48.518417 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.518224 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:48.523476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.523448 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg"] Sep 08 17:07:48.525969 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.525953 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:48.528446 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.528424 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:07:48.528698 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.528681 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:07:48.536930 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.536906 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg"] Sep 08 17:07:48.610151 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.610108 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg\" (UID: \"ee2a772d-b52e-4ae7-9361-586012b02428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:48.711365 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.711306 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg\" (UID: \"ee2a772d-b52e-4ae7-9361-586012b02428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:48.711927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.711908 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg\" (UID: \"ee2a772d-b52e-4ae7-9361-586012b02428\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:48.837193 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:48.837116 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:49.131259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.131232 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg"] Sep 08 17:07:49.133122 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:07:49.133093 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2a772d_b52e_4ae7_9361_586012b02428.slice/crio-ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714 WatchSource:0}: Error finding container ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714: Status 404 returned error can't find the container with id ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714 Sep 08 17:07:49.215071 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.215037 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:49.215387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.215307 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" podUID="054bd77b-1916-4888-a7fc-b4cc21658063" containerName="istio-proxy" containerID="cri-o://67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f" gracePeriod=30 Sep 08 17:07:49.480970 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.480929 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" event={"ID":"ee2a772d-b52e-4ae7-9361-586012b02428","Type":"ContainerStarted","Data":"88eb82088834ab688e57cd1e52df2381a4aa2f6ba760abe5d44a9e6fa42af9ee"} Sep 08 17:07:49.481166 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.480994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" event={"ID":"ee2a772d-b52e-4ae7-9361-586012b02428","Type":"ContainerStarted","Data":"ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714"} Sep 08 17:07:49.499925 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.499871 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" podStartSLOduration=1.499854125 podStartE2EDuration="1.499854125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:49.498600202 +0000 UTC m=+1000.643370764" watchObservedRunningTime="2026-09-08 17:07:49.499854125 +0000 UTC m=+1000.644624680" Sep 08 17:07:49.518613 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.518575 2556 patch_prober.go:28] interesting pod/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 08 17:07:49.518767 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:49.518640 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 08 17:07:50.487910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:50.487874 2556 generic.go:356] "Generic (PLEG): container finished" podID="ee2a772d-b52e-4ae7-9361-586012b02428" containerID="88eb82088834ab688e57cd1e52df2381a4aa2f6ba760abe5d44a9e6fa42af9ee" exitCode=0 Sep 08 17:07:50.487910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:50.487905 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" event={"ID":"ee2a772d-b52e-4ae7-9361-586012b02428","Type":"ContainerDied","Data":"88eb82088834ab688e57cd1e52df2381a4aa2f6ba760abe5d44a9e6fa42af9ee"} Sep 08 17:07:50.522212 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:50.522184 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:50.522557 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:50.522539 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:50.523252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:50.523232 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:07:51.626004 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:51.625977 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:51.742570 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:51.742536 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca\") pod \"ee2a772d-b52e-4ae7-9361-586012b02428\" (UID: \"ee2a772d-b52e-4ae7-9361-586012b02428\") " Sep 08 17:07:51.742897 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:51.742870 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee2a772d-b52e-4ae7-9361-586012b02428" (UID: "ee2a772d-b52e-4ae7-9361-586012b02428"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:07:51.844165 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:51.844075 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2a772d-b52e-4ae7-9361-586012b02428-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:52.103708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:52.103614 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:07:52.500396 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:52.500365 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" Sep 08 17:07:52.500396 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:52.500384 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg" event={"ID":"ee2a772d-b52e-4ae7-9361-586012b02428","Type":"ContainerDied","Data":"ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714"} Sep 08 17:07:52.500605 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:52.500442 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea973441f45d6992f60ea95253c216108c73bd8402797fcfde7fba1fbbb02714" Sep 08 17:07:54.463260 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.463231 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:54.512257 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.512179 2556 generic.go:356] "Generic (PLEG): container finished" podID="054bd77b-1916-4888-a7fc-b4cc21658063" containerID="67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f" exitCode=0 Sep 08 17:07:54.512257 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.512232 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" Sep 08 17:07:54.512257 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.512237 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" event={"ID":"054bd77b-1916-4888-a7fc-b4cc21658063","Type":"ContainerDied","Data":"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f"} Sep 08 17:07:54.512511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.512278 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l" event={"ID":"054bd77b-1916-4888-a7fc-b4cc21658063","Type":"ContainerDied","Data":"1420fd451887ebd9a2913f20ca3b59b77b744d1a915e71768222e672e4cae8ec"} Sep 08 17:07:54.512511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.512301 2556 scope.go:117] "RemoveContainer" containerID="67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f" Sep 08 17:07:54.521199 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.521182 2556 scope.go:117] "RemoveContainer" containerID="67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f" Sep 08 17:07:54.521457 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:07:54.521439 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f\": container with ID starting with 67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f not found: ID does not exist" containerID="67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f" Sep 08 17:07:54.521508 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.521466 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f"} err="failed to get container status \"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f\": rpc error: code = NotFound desc = could not find container \"67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f\": container with ID starting with 67cfb0d191302bbe17e8be34692dca1eee6dfb05b0cc6fe06ff994287fb7715f not found: ID does not exist" Sep 08 17:07:54.572493 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572460 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572504 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmqt8\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572558 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572583 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572634 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572604 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572858 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572643 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572858 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572697 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572858 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572775 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.572858 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data\") pod \"054bd77b-1916-4888-a7fc-b4cc21658063\" (UID: \"054bd77b-1916-4888-a7fc-b4cc21658063\") " Sep 08 17:07:54.573076 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572767 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:54.573076 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.572957 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:54.573181 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573145 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data" (OuterVolumeSpecName: "istio-data") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:54.573181 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573160 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:54.573274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573235 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.573274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573255 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.573274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573269 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.573425 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.573316 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:07:54.574826 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.574798 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8" (OuterVolumeSpecName: "kube-api-access-rmqt8") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "kube-api-access-rmqt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:07:54.574933 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.574896 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:07:54.575136 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.575113 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:07:54.575136 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.575125 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token" (OuterVolumeSpecName: "istio-token") pod "054bd77b-1916-4888-a7fc-b4cc21658063" (UID: "054bd77b-1916-4888-a7fc-b4cc21658063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:07:54.674542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674506 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqt8\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-kube-api-access-rmqt8\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.674542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674533 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.674542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674543 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054bd77b-1916-4888-a7fc-b4cc21658063-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.674542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674551 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054bd77b-1916-4888-a7fc-b4cc21658063-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.674820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674559 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054bd77b-1916-4888-a7fc-b4cc21658063-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.674820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.674567 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054bd77b-1916-4888-a7fc-b4cc21658063-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:07:54.836339 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.836292 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:54.839210 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:54.839170 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-79e919-openshift-default-67cc7ccc88-b6b9l"] Sep 08 17:07:55.343011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:55.342978 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054bd77b-1916-4888-a7fc-b4cc21658063" path="/var/lib/kubelet/pods/054bd77b-1916-4888-a7fc-b4cc21658063/volumes" Sep 08 17:07:57.728800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.728766 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm"] Sep 08 17:07:57.729195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729182 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054bd77b-1916-4888-a7fc-b4cc21658063" containerName="istio-proxy" Sep 08 17:07:57.729195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729194 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="054bd77b-1916-4888-a7fc-b4cc21658063" containerName="istio-proxy" Sep 08 17:07:57.729264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729211 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2a772d-b52e-4ae7-9361-586012b02428" containerName="revoke-keys" Sep 08 17:07:57.729264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729217 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2a772d-b52e-4ae7-9361-586012b02428" containerName="revoke-keys" Sep 08 17:07:57.729346 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729285 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2a772d-b52e-4ae7-9361-586012b02428" containerName="revoke-keys" Sep 08 17:07:57.729346 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.729294 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="054bd77b-1916-4888-a7fc-b4cc21658063" containerName="istio-proxy" Sep 08 17:07:57.733786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.733764 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:07:57.739034 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.739008 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:07:57.739184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.739166 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:07:57.747048 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.747023 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm"] Sep 08 17:07:57.805493 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.805449 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm\" (UID: \"22b00d55-018c-4b94-8b62-a731e2e42046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:07:57.907094 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.907056 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm\" (UID: \"22b00d55-018c-4b94-8b62-a731e2e42046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:07:57.907662 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:57.907645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm\" (UID: \"22b00d55-018c-4b94-8b62-a731e2e42046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:07:58.045373 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:58.045266 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:07:58.319006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:58.318921 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm"] Sep 08 17:07:58.321888 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:07:58.321857 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b00d55_018c_4b94_8b62_a731e2e42046.slice/crio-0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642 WatchSource:0}: Error finding container 0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642: Status 404 returned error can't find the container with id 0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642 Sep 08 17:07:58.536050 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:58.536017 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" event={"ID":"22b00d55-018c-4b94-8b62-a731e2e42046","Type":"ContainerStarted","Data":"0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642"} Sep 08 17:07:59.099400 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:59.099367 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg"] Sep 08 17:07:59.102388 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:59.102360 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2746ad-qnqgg"] Sep 08 17:07:59.344068 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:59.343989 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2a772d-b52e-4ae7-9361-586012b02428" path="/var/lib/kubelet/pods/ee2a772d-b52e-4ae7-9361-586012b02428/volumes" Sep 08 17:07:59.542578 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:59.542538 2556 generic.go:356] "Generic (PLEG): container finished" podID="22b00d55-018c-4b94-8b62-a731e2e42046" containerID="96e43b3e3bd107849cce440449517f270af457385fb8674679201e4879c79584" exitCode=0 Sep 08 17:07:59.542747 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:07:59.542619 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" event={"ID":"22b00d55-018c-4b94-8b62-a731e2e42046","Type":"ContainerDied","Data":"96e43b3e3bd107849cce440449517f270af457385fb8674679201e4879c79584"} Sep 08 17:08:00.674021 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:00.673989 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:08:00.736998 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:00.736961 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca\") pod \"22b00d55-018c-4b94-8b62-a731e2e42046\" (UID: \"22b00d55-018c-4b94-8b62-a731e2e42046\") " Sep 08 17:08:00.737308 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:00.737283 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22b00d55-018c-4b94-8b62-a731e2e42046" (UID: "22b00d55-018c-4b94-8b62-a731e2e42046"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:00.837954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:00.837867 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22b00d55-018c-4b94-8b62-a731e2e42046-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:01.555393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:01.555360 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" Sep 08 17:08:01.555393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:01.555384 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm" event={"ID":"22b00d55-018c-4b94-8b62-a731e2e42046","Type":"ContainerDied","Data":"0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642"} Sep 08 17:08:01.555598 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:01.555431 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a28f570e0cf9e79e013f3e5d1a857f0acf5d6f29af18523eac46aa45aaf6642" Sep 08 17:08:02.308498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:02.308462 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:08:05.567085 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:05.567052 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:08:05.567580 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:05.567377 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" containerID="cri-o://ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97" gracePeriod=30 Sep 08 17:08:08.488980 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.488941 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:08:08.489401 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.489180 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" podUID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" containerName="payload-pre-processing" containerID="cri-o://e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f" gracePeriod=30 Sep 08 17:08:08.651991 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.651959 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm"] Sep 08 17:08:08.656408 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.656381 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a837a8a9-m4qbm"] Sep 08 17:08:08.749865 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.749802 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:08:08.921784 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.921747 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume\") pod \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " Sep 08 17:08:08.921961 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.921812 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv6ff\" (UniqueName: \"kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff\") pod \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\" (UID: \"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2\") " Sep 08 17:08:08.922204 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.922177 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" (UID: "7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:08.923927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:08.923904 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff" (OuterVolumeSpecName: "kube-api-access-dv6ff") pod "7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" (UID: "7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2"). InnerVolumeSpecName "kube-api-access-dv6ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:09.022636 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.022540 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6ff\" (UniqueName: \"kubernetes.io/projected/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-kube-api-access-dv6ff\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:09.022636 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.022574 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:09.344208 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.344134 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b00d55-018c-4b94-8b62-a731e2e42046" path="/var/lib/kubelet/pods/22b00d55-018c-4b94-8b62-a731e2e42046/volumes" Sep 08 17:08:09.604494 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.604405 2556 generic.go:356] "Generic (PLEG): container finished" podID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" containerID="e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f" exitCode=0 Sep 08 17:08:09.604494 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.604464 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" Sep 08 17:08:09.604924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.604492 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" event={"ID":"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2","Type":"ContainerDied","Data":"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f"} Sep 08 17:08:09.604924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.604549 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm" event={"ID":"7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2","Type":"ContainerDied","Data":"8c62fa5a53a6b4f1e5b2795c0dcb04089c360650a020e85e02933561a4892ce2"} Sep 08 17:08:09.604924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.604575 2556 scope.go:117] "RemoveContainer" containerID="e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f" Sep 08 17:08:09.613213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.613191 2556 scope.go:117] "RemoveContainer" containerID="e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f" Sep 08 17:08:09.613496 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:09.613477 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f\": container with ID starting with e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f not found: ID does not exist" containerID="e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f" Sep 08 17:08:09.613564 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.613505 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f"} err="failed to get container status \"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f\": rpc error: code = NotFound desc = could not find container \"e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f\": container with ID starting with e1cab374749724f024d8c5cb10fd8dc598acaf17d81f1d2c9075953235def96f not found: ID does not exist" Sep 08 17:08:09.620905 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.620879 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:08:09.622676 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:09.622655 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a837a8a9-56cbb89854-86vlm"] Sep 08 17:08:10.465715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.465681 2556 patch_prober.go:28] interesting pod/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:08:10.465891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.465741 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:08:10.821223 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.821196 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:08:10.939646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939603 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z42rs\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.939845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939663 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.939845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939695 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.939845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939796 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.939845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939839 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.940062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939863 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.940062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939918 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.940062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939970 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.940062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939998 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket\") pod \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\" (UID: \"afb70d0c-dfde-4ee4-a80d-8f717beffc7a\") " Sep 08 17:08:10.940249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.939962 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:10.940249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940087 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:10.940249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940165 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:10.940448 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940344 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:10.940448 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940402 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:10.940448 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940442 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:10.940566 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940457 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:10.940566 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940465 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:10.940566 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.940475 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:10.942393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.942368 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:10.942393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.942381 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs" (OuterVolumeSpecName: "kube-api-access-z42rs") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "kube-api-access-z42rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:10.942511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.942483 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:10.942511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:10.942487 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afb70d0c-dfde-4ee4-a80d-8f717beffc7a" (UID: "afb70d0c-dfde-4ee4-a80d-8f717beffc7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:08:11.041825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.041742 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:11.041825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.041769 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42rs\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-kube-api-access-z42rs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:11.041825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.041779 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:11.041825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.041787 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:11.041825 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.041794 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb70d0c-dfde-4ee4-a80d-8f717beffc7a-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:11.343347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.343256 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" path="/var/lib/kubelet/pods/7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2/volumes" Sep 08 17:08:11.618508 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.618479 2556 generic.go:356] "Generic (PLEG): container finished" podID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerID="ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97" exitCode=0 Sep 08 17:08:11.618655 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.618530 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" Sep 08 17:08:11.618655 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.618570 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerDied","Data":"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97"} Sep 08 17:08:11.618655 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.618622 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc" event={"ID":"afb70d0c-dfde-4ee4-a80d-8f717beffc7a","Type":"ContainerDied","Data":"7e5e82d4367ad5361efdce1a92fb3e7c8a19b8a9cc58451c08c6291958da1bdf"} Sep 08 17:08:11.618655 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.618644 2556 scope.go:117] "RemoveContainer" containerID="ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97" Sep 08 17:08:11.627619 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.627605 2556 scope.go:117] "RemoveContainer" containerID="570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd" Sep 08 17:08:11.636189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.636172 2556 scope.go:117] "RemoveContainer" containerID="ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97" Sep 08 17:08:11.636500 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:11.636480 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97\": container with ID starting with ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97 not found: ID does not exist" containerID="ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97" Sep 08 17:08:11.636545 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.636509 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97"} err="failed to get container status \"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97\": rpc error: code = NotFound desc = could not find container \"ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97\": container with ID starting with ad25bacd865a616c0229bb5a62f3c63aba70572f60b4be48e2dea08fcf3d3a97 not found: ID does not exist" Sep 08 17:08:11.636545 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.636524 2556 scope.go:117] "RemoveContainer" containerID="570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd" Sep 08 17:08:11.636804 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:11.636782 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd\": container with ID starting with 570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd not found: ID does not exist" containerID="570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd" Sep 08 17:08:11.636860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.636813 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd"} err="failed to get container status \"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd\": rpc error: code = NotFound desc = could not find container \"570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd\": container with ID starting with 570efdeb14afd9de7ed3e2c815a53f4761edb279252ce06705c14167d6d399fd not found: ID does not exist" Sep 08 17:08:11.637416 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.637397 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:08:11.642670 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:11.642651 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2746ad-openshift-default-5565675cf4-m6wvc"] Sep 08 17:08:13.343626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:13.343581 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" path="/var/lib/kubelet/pods/afb70d0c-dfde-4ee4-a80d-8f717beffc7a/volumes" Sep 08 17:08:13.698003 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:13.697965 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:08:14.997433 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:14.997398 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:08:14.997795 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:14.997689 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" containerID="cri-o://48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6" gracePeriod=30 Sep 08 17:08:16.769349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:16.769299 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:08:16.769799 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:16.769583 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" podUID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" containerName="payload-processing" containerID="cri-o://879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984" gracePeriod=30 Sep 08 17:08:17.024827 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.024763 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:08:17.201942 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.201902 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume\") pod \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " Sep 08 17:08:17.202114 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.201971 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct7hp\" (UniqueName: \"kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp\") pod \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\" (UID: \"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae\") " Sep 08 17:08:17.202298 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.202266 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" (UID: "ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:17.204025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.203999 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp" (OuterVolumeSpecName: "kube-api-access-ct7hp") pod "ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" (UID: "ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae"). InnerVolumeSpecName "kube-api-access-ct7hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:17.303485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.303395 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.303485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.303423 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7hp\" (UniqueName: \"kubernetes.io/projected/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae-kube-api-access-ct7hp\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:17.652032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.651995 2556 generic.go:356] "Generic (PLEG): container finished" podID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" containerID="879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984" exitCode=0 Sep 08 17:08:17.652236 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.652042 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" Sep 08 17:08:17.652236 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.652066 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" event={"ID":"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae","Type":"ContainerDied","Data":"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984"} Sep 08 17:08:17.652236 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.652107 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl" event={"ID":"ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae","Type":"ContainerDied","Data":"4e9165599a1295f7e5bedf573bb5842102a2197ebb07bdb513745e92e945bd89"} Sep 08 17:08:17.652236 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.652128 2556 scope.go:117] "RemoveContainer" containerID="879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984" Sep 08 17:08:17.661527 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.661500 2556 scope.go:117] "RemoveContainer" containerID="879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984" Sep 08 17:08:17.661832 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:17.661813 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984\": container with ID starting with 879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984 not found: ID does not exist" containerID="879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984" Sep 08 17:08:17.661900 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.661841 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984"} err="failed to get container status \"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984\": rpc error: code = NotFound desc = could not find container \"879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984\": container with ID starting with 879b942d209278d40b9484eb2140efe248eddefc71e86286b5017c1417002984 not found: ID does not exist" Sep 08 17:08:17.676339 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.672700 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:08:17.681288 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:17.681261 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7cab-dffcf8565-qlknl"] Sep 08 17:08:19.342246 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:19.342206 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" path="/var/lib/kubelet/pods/ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae/volumes" Sep 08 17:08:20.243162 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.243133 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:08:20.329227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329140 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329187 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329206 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329228 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329255 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329292 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329318 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjctm\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329411 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329452 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket\") pod \"1f75ebf7-46aa-46a1-9443-a9994194e78b\" (UID: \"1f75ebf7-46aa-46a1-9443-a9994194e78b\") " Sep 08 17:08:20.329618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329538 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:20.329904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329613 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:20.329904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329628 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:20.329904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329824 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:20.329904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329884 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.330065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329920 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.330065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329924 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:20.330065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.329936 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.331629 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.331600 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:20.331731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.331646 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm" (OuterVolumeSpecName: "kube-api-access-gjctm") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "kube-api-access-gjctm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:20.331840 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.331822 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:08:20.331880 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.331823 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f75ebf7-46aa-46a1-9443-a9994194e78b" (UID: "1f75ebf7-46aa-46a1-9443-a9994194e78b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:20.430640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430590 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.430640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430631 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.430640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430641 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.430640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430651 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f75ebf7-46aa-46a1-9443-a9994194e78b-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.430640 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430659 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjctm\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-kube-api-access-gjctm\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.431139 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.430668 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f75ebf7-46aa-46a1-9443-a9994194e78b-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:20.669130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.669091 2556 generic.go:356] "Generic (PLEG): container finished" podID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerID="48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6" exitCode=0 Sep 08 17:08:20.669349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.669142 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" Sep 08 17:08:20.669349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.669171 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerDied","Data":"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6"} Sep 08 17:08:20.669349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.669230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn" event={"ID":"1f75ebf7-46aa-46a1-9443-a9994194e78b","Type":"ContainerDied","Data":"f315f2af332ac865e9ab1fe721177ffaa87b881772242acfefabe3d32ddf2367"} Sep 08 17:08:20.669349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.669252 2556 scope.go:117] "RemoveContainer" containerID="48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6" Sep 08 17:08:20.678797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.678782 2556 scope.go:117] "RemoveContainer" containerID="375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a" Sep 08 17:08:20.686591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.686573 2556 scope.go:117] "RemoveContainer" containerID="48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6" Sep 08 17:08:20.686870 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:20.686851 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6\": container with ID starting with 48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6 not found: ID does not exist" containerID="48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6" Sep 08 17:08:20.686918 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.686878 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6"} err="failed to get container status \"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6\": rpc error: code = NotFound desc = could not find container \"48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6\": container with ID starting with 48232c25c02f99e1c4004964a70c4657fefb1971dc89029a988859e8e3470ce6 not found: ID does not exist" Sep 08 17:08:20.686918 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.686895 2556 scope.go:117] "RemoveContainer" containerID="375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a" Sep 08 17:08:20.687144 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:20.687126 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a\": container with ID starting with 375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a not found: ID does not exist" containerID="375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a" Sep 08 17:08:20.687190 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.687149 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a"} err="failed to get container status \"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a\": rpc error: code = NotFound desc = could not find container \"375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a\": container with ID starting with 375eb96519e7e9bea1df90a10c04d74aa0a4e91d9e45360bb02214c459e7190a not found: ID does not exist" Sep 08 17:08:20.690291 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.690272 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:08:20.694000 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:20.693980 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a837a8a9-openshift-default-74f887cd88-fxvvn"] Sep 08 17:08:21.342979 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:21.342948 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" path="/var/lib/kubelet/pods/1f75ebf7-46aa-46a1-9443-a9994194e78b/volumes" Sep 08 17:08:21.704242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:21.704205 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:08:23.807534 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:23.807504 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:08:23.807961 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:23.807741 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" containerID="cri-o://f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b" gracePeriod=30 Sep 08 17:08:24.666376 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666338 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:08:24.666921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666903 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" containerName="payload-processing" Sep 08 17:08:24.666977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666926 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" containerName="payload-processing" Sep 08 17:08:24.666977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666947 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:24.666977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666956 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:24.666977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666968 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.666977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666978 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666993 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.666999 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667007 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" containerName="payload-pre-processing" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667013 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" containerName="payload-pre-processing" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667029 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b00d55-018c-4b94-8b62-a731e2e42046" containerName="revoke-keys" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667034 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b00d55-018c-4b94-8b62-a731e2e42046" containerName="revoke-keys" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667120 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.667128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667126 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7a3d4a-4a0d-41a4-a33a-8d0f87bc9eae" containerName="payload-processing" Sep 08 17:08:24.667387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667132 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f93ee78-30ec-4c1a-b1e4-e4b13f40caf2" containerName="payload-pre-processing" Sep 08 17:08:24.667387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667138 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b00d55-018c-4b94-8b62-a731e2e42046" containerName="revoke-keys" Sep 08 17:08:24.667387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667146 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:24.667387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667152 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb70d0c-dfde-4ee4-a80d-8f717beffc7a" containerName="istio-proxy" Sep 08 17:08:24.667387 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.667158 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:24.671741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.671722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.681824 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.681808 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c05c9f-openshift-default-dockercfg-qvn2k\"" Sep 08 17:08:24.718503 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.718463 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:08:24.772374 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772430 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772469 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772716 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.772849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.772820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvqw\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873311 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873275 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvqw\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873348 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873380 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873487 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873544 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.873832 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.874281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873883 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.874281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.873962 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.874281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.874063 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.874281 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.874107 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.874512 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.874370 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.875904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.875881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.875982 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.875881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.891344 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.891298 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.893757 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.893734 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvqw\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw\") pod \"e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:24.981431 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:24.981313 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:25.259479 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.259449 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:08:25.262016 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:25.261989 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd354ff_1818_430c_9abd_0d4135f952ff.slice/crio-bc9f9f86b9e8b88b22c473d42705e533d2d09e87478488612f066c8dc195a047 WatchSource:0}: Error finding container bc9f9f86b9e8b88b22c473d42705e533d2d09e87478488612f066c8dc195a047: Status 404 returned error can't find the container with id bc9f9f86b9e8b88b22c473d42705e533d2d09e87478488612f066c8dc195a047 Sep 08 17:08:25.264103 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.264076 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:25.264174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.264139 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:25.264174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.264169 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:25.697544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.697506 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerStarted","Data":"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993"} Sep 08 17:08:25.697701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.697566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerStarted","Data":"bc9f9f86b9e8b88b22c473d42705e533d2d09e87478488612f066c8dc195a047"} Sep 08 17:08:25.719952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.719899 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podStartSLOduration=1.719884275 podStartE2EDuration="1.719884275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:25.716983227 +0000 UTC m=+1036.861753787" watchObservedRunningTime="2026-09-08 17:08:25.719884275 +0000 UTC m=+1036.864654836" Sep 08 17:08:25.982233 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:25.982141 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:26.986676 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:26.986645 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:27.713255 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:27.713219 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:27.714359 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:27.714337 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:29.064160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.064131 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:08:29.118366 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118337 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118546 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118415 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118546 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118513 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118600 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118723 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118678 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118782 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118718 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118782 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118758 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118779 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:08:29.118891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118798 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118830 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhsg5\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5\") pod \"bd645f7b-02fd-44c7-abfe-483285abd5aa\" (UID: \"bd645f7b-02fd-44c7-abfe-483285abd5aa\") " Sep 08 17:08:29.118891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118881 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:29.119107 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.118953 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:29.119169 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.119150 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:29.119229 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.119175 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.119229 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.119194 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd645f7b-02fd-44c7-abfe-483285abd5aa-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.119229 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.119207 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.119425 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.119404 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:29.120962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.120933 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5" (OuterVolumeSpecName: "kube-api-access-lhsg5") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "kube-api-access-lhsg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:29.121078 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.121035 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:08:29.121141 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.121120 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:08:29.121247 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.121229 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd645f7b-02fd-44c7-abfe-483285abd5aa" (UID: "bd645f7b-02fd-44c7-abfe-483285abd5aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:08:29.220507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220473 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.220507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220502 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.220507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220512 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.220748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220520 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhsg5\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-kube-api-access-lhsg5\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.220748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220530 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.220748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.220538 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd645f7b-02fd-44c7-abfe-483285abd5aa-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:08:29.468603 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.468512 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:08:29.469062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469044 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:29.469130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469065 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:29.469130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469094 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:29.469130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469104 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f75ebf7-46aa-46a1-9443-a9994194e78b" containerName="istio-proxy" Sep 08 17:08:29.469130 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469124 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:29.469258 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469133 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:29.469258 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.469216 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:29.472833 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.472810 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.476458 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.476436 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-ec96ff-openshift-default-dockercfg-4q45z\"" Sep 08 17:08:29.488100 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.488064 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:08:29.523376 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523340 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523376 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523376 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523514 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523569 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523600 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523798 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523632 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx89\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523798 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523691 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.523798 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.523776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624629 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624593 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624672 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624708 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.624801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.624752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625085 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625062 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625114 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx89\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625134 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625147 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625208 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625455 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.625772 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.625752 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.627113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.627097 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.627292 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.627275 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.637157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.637132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.637868 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.637853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx89\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89\") pod \"e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:29.727568 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.727491 2556 generic.go:356] "Generic (PLEG): container finished" podID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerID="f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b" exitCode=0 Sep 08 17:08:29.727568 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.727550 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" Sep 08 17:08:29.727748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.727568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerDied","Data":"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b"} Sep 08 17:08:29.727748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.727625 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k" event={"ID":"bd645f7b-02fd-44c7-abfe-483285abd5aa","Type":"ContainerDied","Data":"8557cba974ec587e2ca65a24221a5ac34b3be2deca338a883efaf1f5e2cf6ab8"} Sep 08 17:08:29.727748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.727647 2556 scope.go:117] "RemoveContainer" containerID="f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b" Sep 08 17:08:29.736677 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.736656 2556 scope.go:117] "RemoveContainer" containerID="4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7" Sep 08 17:08:29.746155 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.746136 2556 scope.go:117] "RemoveContainer" containerID="f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b" Sep 08 17:08:29.746444 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:29.746425 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b\": container with ID starting with f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b not found: ID does not exist" containerID="f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b" Sep 08 17:08:29.746496 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.746451 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b"} err="failed to get container status \"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b\": rpc error: code = NotFound desc = could not find container \"f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b\": container with ID starting with f3ebb4b588485e0aa2f957a41f38bce67a223d5ff745da866754e74aaeb1eb5b not found: ID does not exist" Sep 08 17:08:29.746496 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.746468 2556 scope.go:117] "RemoveContainer" containerID="4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7" Sep 08 17:08:29.746730 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:08:29.746713 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7\": container with ID starting with 4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7 not found: ID does not exist" containerID="4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7" Sep 08 17:08:29.746771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.746733 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7"} err="failed to get container status \"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7\": rpc error: code = NotFound desc = could not find container \"4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7\": container with ID starting with 4efac2032d096e27fdc7ac9b811fb4f1a286ec9b4f6b23303f26a8530cca14f7 not found: ID does not exist" Sep 08 17:08:29.759613 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.759588 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:08:29.763867 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.763847 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7cab-openshift-default-698d86b654-tv75k"] Sep 08 17:08:29.784156 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:29.784135 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:30.063859 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.063705 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:08:30.066541 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:30.066510 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de5f12b_8360_4eed_98a2_a557d549bc05.slice/crio-7806333db68f3c134db606749b5fd25b40862a1b7f63a2aa62401e0f25d8abe5 WatchSource:0}: Error finding container 7806333db68f3c134db606749b5fd25b40862a1b7f63a2aa62401e0f25d8abe5: Status 404 returned error can't find the container with id 7806333db68f3c134db606749b5fd25b40862a1b7f63a2aa62401e0f25d8abe5 Sep 08 17:08:30.069147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.069114 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:30.069230 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.069177 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:30.069230 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.069206 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:30.734597 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.734554 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerStarted","Data":"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61"} Sep 08 17:08:30.734798 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.734620 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerStarted","Data":"7806333db68f3c134db606749b5fd25b40862a1b7f63a2aa62401e0f25d8abe5"} Sep 08 17:08:30.758736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.758687 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podStartSLOduration=1.7586725479999998 podStartE2EDuration="1.758672548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:30.757056662 +0000 UTC m=+1041.901827224" watchObservedRunningTime="2026-09-08 17:08:30.758672548 +0000 UTC m=+1041.903443182" Sep 08 17:08:30.785037 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:30.785004 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:31.343148 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:31.343115 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" path="/var/lib/kubelet/pods/bd645f7b-02fd-44c7-abfe-483285abd5aa/volumes" Sep 08 17:08:31.789937 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:31.789908 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:32.745900 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:32.745870 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:32.747015 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:32.746995 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:32.844762 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:32.844730 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:08:34.556888 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.556846 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:08:34.557492 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.557472 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd645f7b-02fd-44c7-abfe-483285abd5aa" containerName="istio-proxy" Sep 08 17:08:34.561697 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.561677 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.564601 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.564575 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ec96ff-dockercfg-ck5z7\"" Sep 08 17:08:34.564724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.564602 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ec96ff\"" Sep 08 17:08:34.574469 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.574438 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:08:34.581585 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.581557 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:08:34.585220 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.585199 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.601086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.601057 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:08:34.675868 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.675827 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl5ft\" (UniqueName: \"kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.676088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.676047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.676303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.676285 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.676545 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.676529 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkbd\" (UniqueName: \"kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.777952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.777914 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl5ft\" (UniqueName: \"kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.778153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.777967 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.778153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.777995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.778153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.778043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkbd\" (UniqueName: \"kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.778849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.778823 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.778849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.778838 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.785917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.785886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkbd\" (UniqueName: \"kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd\") pod \"payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:34.786021 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.785934 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl5ft\" (UniqueName: \"kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft\") pod \"payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.872786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.872738 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:34.904807 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:34.904768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:35.151398 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.149957 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:08:35.167514 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:35.167483 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900beb2f_d926_4cad_b526_911caba07887.slice/crio-0cea311818e7f1d89eb9d2580d28d5c4b9b818227f08e48def052ccb46d2e853 WatchSource:0}: Error finding container 0cea311818e7f1d89eb9d2580d28d5c4b9b818227f08e48def052ccb46d2e853: Status 404 returned error can't find the container with id 0cea311818e7f1d89eb9d2580d28d5c4b9b818227f08e48def052ccb46d2e853 Sep 08 17:08:35.217887 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.217864 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:08:35.222527 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:35.222499 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9d803a_5338_482f_8d66_c774b82108c9.slice/crio-2178d29a7556b16bc7b2bae71f39545758856dddec298e56bc536e39151b6131 WatchSource:0}: Error finding container 2178d29a7556b16bc7b2bae71f39545758856dddec298e56bc536e39151b6131: Status 404 returned error can't find the container with id 2178d29a7556b16bc7b2bae71f39545758856dddec298e56bc536e39151b6131 Sep 08 17:08:35.770956 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.770914 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" event={"ID":"2c9d803a-5338-482f-8d66-c774b82108c9","Type":"ContainerStarted","Data":"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363"} Sep 08 17:08:35.771485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.770981 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" event={"ID":"2c9d803a-5338-482f-8d66-c774b82108c9","Type":"ContainerStarted","Data":"2178d29a7556b16bc7b2bae71f39545758856dddec298e56bc536e39151b6131"} Sep 08 17:08:35.771778 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.771755 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:35.773962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.773934 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" event={"ID":"900beb2f-d926-4cad-b526-911caba07887","Type":"ContainerStarted","Data":"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f"} Sep 08 17:08:35.774077 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.773986 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" event={"ID":"900beb2f-d926-4cad-b526-911caba07887","Type":"ContainerStarted","Data":"0cea311818e7f1d89eb9d2580d28d5c4b9b818227f08e48def052ccb46d2e853"} Sep 08 17:08:35.774471 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.774421 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:35.789723 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.789669 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" podStartSLOduration=1.7896477910000002 podStartE2EDuration="1.789647791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:35.789280852 +0000 UTC m=+1046.934051416" watchObservedRunningTime="2026-09-08 17:08:35.789647791 +0000 UTC m=+1046.934418355" Sep 08 17:08:35.806904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:35.806852 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" podStartSLOduration=1.806839327 podStartE2EDuration="1.806839327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:35.804932052 +0000 UTC m=+1046.949702606" watchObservedRunningTime="2026-09-08 17:08:35.806839327 +0000 UTC m=+1046.951609888" Sep 08 17:08:41.911022 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:41.910986 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:08:43.590224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.589005 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:51860->10.134.0.54:15021: read: connection reset by peer" start-of-body= Sep 08 17:08:43.590224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.589070 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:51860->10.134.0.54:15021: read: connection reset by peer" Sep 08 17:08:43.823145 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.823058 2556 generic.go:356] "Generic (PLEG): container finished" podID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerID="4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993" exitCode=0 Sep 08 17:08:43.823302 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.823139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerDied","Data":"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993"} Sep 08 17:08:43.823601 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.823585 2556 scope.go:117] "RemoveContainer" containerID="4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993" Sep 08 17:08:43.824618 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.824589 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:43.824667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.824658 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:43.824707 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:43.824685 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:44.831397 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:44.831355 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerStarted","Data":"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f"} Sep 08 17:08:44.981723 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:44.981680 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:44.981922 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:44.981899 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:44.981995 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:44.981960 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:45.981748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:45.981711 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:45.982138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:45.981771 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:46.782046 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:46.782016 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:08:46.981919 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:46.981878 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:46.982295 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:46.981935 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:47.787488 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:47.787459 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:08:47.981904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:47.981861 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:47.982086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:47.981924 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:48.274706 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.274545 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:44462->10.134.0.55:15021: read: connection reset by peer" start-of-body= Sep 08 17:08:48.274706 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.274608 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:44462->10.134.0.55:15021: read: connection reset by peer" Sep 08 17:08:48.867228 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.867195 2556 generic.go:356] "Generic (PLEG): container finished" podID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerID="bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61" exitCode=0 Sep 08 17:08:48.867473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.867438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerDied","Data":"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61"} Sep 08 17:08:48.867971 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.867949 2556 scope.go:117] "RemoveContainer" containerID="bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61" Sep 08 17:08:48.869307 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.869272 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:48.869405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.869384 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:48.869459 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.869442 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:48.982669 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.982628 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:48.983165 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:48.982690 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:49.880691 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:49.880630 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerStarted","Data":"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4"} Sep 08 17:08:49.982445 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:49.982397 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 17:08:49.982659 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:49.982472 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 17:08:50.784661 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.784623 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:50.785059 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.784833 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 17:08:50.785059 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.784886 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 17:08:50.985764 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.985734 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:50.986060 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.986038 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:50.986650 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:50.986634 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:08:51.785257 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:51.785218 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 17:08:51.785670 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:51.785278 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 17:08:52.784542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:52.784502 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 17:08:52.784718 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:52.784562 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 17:08:53.158348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.157625 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:08:53.167597 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.167572 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.172891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.172866 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-338600-openshift-default-dockercfg-rqm9c\"" Sep 08 17:08:53.174038 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.174016 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.278948 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279041 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279086 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjzt\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279119 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279195 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279261 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.281348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.279300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380689 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380724 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380752 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjzt\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380780 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.380929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380907 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381172 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.380945 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381172 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381037 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381407 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381381 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381522 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381522 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381470 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381522 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.381693 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.381659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.383071 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.383049 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.383316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.383299 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.395593 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.395569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjzt\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.397990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.397968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token\") pod \"e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.485141 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.485055 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:53.769757 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.769724 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:08:53.772071 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:53.772042 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076f4b03_485d_4b89_bcf9_c6aebdcdea83.slice/crio-08ede787b8fa71ed16b61a2092747bcfb599614d897c16490a56afa1e81a6fb3 WatchSource:0}: Error finding container 08ede787b8fa71ed16b61a2092747bcfb599614d897c16490a56afa1e81a6fb3: Status 404 returned error can't find the container with id 08ede787b8fa71ed16b61a2092747bcfb599614d897c16490a56afa1e81a6fb3 Sep 08 17:08:53.774539 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.774505 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:53.774616 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.774579 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:53.774616 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.774608 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:53.785146 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.785120 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 17:08:53.785248 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.785172 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 17:08:53.905788 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.905758 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerStarted","Data":"9fe349b22ada60d0771f751c741bba5e5efdf18c74a264d0b0d09ced77db9dab"} Sep 08 17:08:53.905956 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.905806 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerStarted","Data":"08ede787b8fa71ed16b61a2092747bcfb599614d897c16490a56afa1e81a6fb3"} Sep 08 17:08:53.928122 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:53.928065 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podStartSLOduration=0.928047495 podStartE2EDuration="928.047495ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:53.926443577 +0000 UTC m=+1065.071214140" watchObservedRunningTime="2026-09-08 17:08:53.928047495 +0000 UTC m=+1065.072818056" Sep 08 17:08:54.486024 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.485986 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:54.491081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.491057 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:54.784818 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.784727 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 17:08:54.784818 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.784787 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 17:08:54.915562 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.915528 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:54.919152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:54.919121 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:08:55.788191 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:55.788158 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:55.788591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:55.788564 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:55.789115 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:55.789097 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:08:58.488062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.488029 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:08:58.493166 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.493135 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.499157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.499131 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d7359b-openshift-default-dockercfg-42cq9\"" Sep 08 17:08:58.533635 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533601 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533782 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533782 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533731 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533863 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533863 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtdd\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533863 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533827 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533915 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533959 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.533993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.533990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.577889 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.577860 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:08:58.634613 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.634780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtdd\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.634780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.634780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.634780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634762 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634882 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.634917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635180 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635016 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635180 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635093 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635313 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635456 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635409 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635456 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.635800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.635780 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.637476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.637447 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.637738 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.637717 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.654586 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.654558 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtdd\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.662067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.662044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token\") pod \"e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:58.805274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:58.805178 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:08:59.110444 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.110418 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:08:59.112180 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:08:59.112153 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1554d2d0_9775_499d_af71_815c1d3a721c.slice/crio-9a96a2ae736d3241761b870718f9cb697497ce7f8b1c6b6c0389b4a49c385f72 WatchSource:0}: Error finding container 9a96a2ae736d3241761b870718f9cb697497ce7f8b1c6b6c0389b4a49c385f72: Status 404 returned error can't find the container with id 9a96a2ae736d3241761b870718f9cb697497ce7f8b1c6b6c0389b4a49c385f72 Sep 08 17:08:59.117285 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.117251 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:59.117428 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.117315 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:59.117428 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.117370 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:08:59.951504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.951465 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerStarted","Data":"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa"} Sep 08 17:08:59.952084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:08:59.951523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerStarted","Data":"9a96a2ae736d3241761b870718f9cb697497ce7f8b1c6b6c0389b4a49c385f72"} Sep 08 17:09:00.005971 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:00.005920 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podStartSLOduration=2.005901913 podStartE2EDuration="2.005901913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:00.002304742 +0000 UTC m=+1071.147075302" watchObservedRunningTime="2026-09-08 17:09:00.005901913 +0000 UTC m=+1071.150672467" Sep 08 17:09:00.805629 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:00.805582 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:00.810809 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:00.810782 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:00.957428 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:00.957396 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:00.958300 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:00.958281 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:03.050431 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:03.050352 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:09:10.956511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:10.956464 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 17:09:10.956952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:10.956547 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 17:09:13.030807 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.030774 2556 generic.go:356] "Generic (PLEG): container finished" podID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerID="9fe349b22ada60d0771f751c741bba5e5efdf18c74a264d0b0d09ced77db9dab" exitCode=0 Sep 08 17:09:13.031237 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.030846 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerDied","Data":"9fe349b22ada60d0771f751c741bba5e5efdf18c74a264d0b0d09ced77db9dab"} Sep 08 17:09:13.031237 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.031226 2556 scope.go:117] "RemoveContainer" containerID="9fe349b22ada60d0771f751c741bba5e5efdf18c74a264d0b0d09ced77db9dab" Sep 08 17:09:13.032345 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.032303 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:13.032461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.032396 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:13.032461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:13.032441 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:14.038039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:14.038002 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerStarted","Data":"a280a45534628d100cc51f96ba9cf664a8576f91313cd4eea58f6b63fa4e1f0d"} Sep 08 17:09:14.486348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:14.486297 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:09:14.486756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:14.486731 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 08 17:09:14.486902 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:14.486879 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 08 17:09:15.028947 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:15.028909 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:09:15.486535 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:15.486487 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 08 17:09:15.487054 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:15.486562 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 08 17:09:16.486512 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:16.486469 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 08 17:09:16.486698 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:16.486548 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 08 17:09:17.486190 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:17.486154 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 08 17:09:17.486405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:17.486230 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 08 17:09:18.485534 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:18.485492 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 08 17:09:18.486005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:18.485560 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 08 17:09:19.489887 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:19.489858 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:09:19.490390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:19.490259 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:09:19.491147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:19.491125 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:09:21.089720 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.089683 2556 generic.go:356] "Generic (PLEG): container finished" podID="1554d2d0-9775-499d-af71-815c1d3a721c" containerID="48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa" exitCode=0 Sep 08 17:09:21.090168 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.089759 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerDied","Data":"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa"} Sep 08 17:09:21.090239 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.090187 2556 scope.go:117] "RemoveContainer" containerID="48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa" Sep 08 17:09:21.091147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.091121 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:21.091199 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.091185 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:21.091234 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:21.091217 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:22.097040 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:22.097009 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerStarted","Data":"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc"} Sep 08 17:09:22.806334 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:22.806289 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:22.806569 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:22.806548 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 17:09:22.806616 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:22.806593 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 17:09:23.805637 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:23.805602 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 17:09:23.806040 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:23.805672 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 17:09:24.805515 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:24.805475 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 17:09:24.805702 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:24.805537 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 17:09:25.806145 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:25.806107 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 17:09:25.806580 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:25.806165 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 17:09:26.805560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:26.805526 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 17:09:26.805736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:26.805581 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 17:09:27.810513 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:27.810481 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:27.810925 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:27.810730 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:27.811392 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:27.811376 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:09:30.019813 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:30.019777 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:09:37.033379 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.033345 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:09:37.036560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.036542 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.040755 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.040731 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb1565be-openshift-default-dockercfg-l99gv\"" Sep 08 17:09:37.078738 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.078703 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:09:37.142334 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142295 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142503 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142503 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142423 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142503 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142463 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142503 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142498 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142643 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142643 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142582 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxs8\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142710 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142643 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.142710 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.142676 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243460 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243425 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243460 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243465 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243682 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxs8\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243682 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243559 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243682 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243686 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243776 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.243849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.244039 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.243926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.244259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.244230 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.244438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.244273 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.244438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.244294 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.244607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.244586 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.246015 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.245993 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.246112 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.246074 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.275945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.275909 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxs8\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.276524 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.276507 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token\") pod \"e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.346890 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.346795 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:37.649945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.649913 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:09:37.651234 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:09:37.651208 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52f3ef4_ac74_4369_8a5c_2c6f8f25df3c.slice/crio-0bedea3c3ebecd8fb4b5506ac321a5da7cdf0c394385d5d209e287011e2a8326 WatchSource:0}: Error finding container 0bedea3c3ebecd8fb4b5506ac321a5da7cdf0c394385d5d209e287011e2a8326: Status 404 returned error can't find the container with id 0bedea3c3ebecd8fb4b5506ac321a5da7cdf0c394385d5d209e287011e2a8326 Sep 08 17:09:37.653446 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.653410 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:37.653526 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.653475 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:37.653526 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:37.653503 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:09:38.187210 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:38.187168 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerStarted","Data":"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c"} Sep 08 17:09:38.187695 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:38.187225 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerStarted","Data":"0bedea3c3ebecd8fb4b5506ac321a5da7cdf0c394385d5d209e287011e2a8326"} Sep 08 17:09:38.222163 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:38.222108 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podStartSLOduration=2.222093652 podStartE2EDuration="2.222093652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:38.220533365 +0000 UTC m=+1109.365303926" watchObservedRunningTime="2026-09-08 17:09:38.222093652 +0000 UTC m=+1109.366864212" Sep 08 17:09:38.348110 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:38.348053 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:38.355904 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:38.355872 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:39.192500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:39.192473 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:39.193367 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:39.193345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:09:42.554176 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.554140 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:09:42.557857 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.557841 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.560577 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.560550 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb1565be\"" Sep 08 17:09:42.560705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.560660 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb1565be-dockercfg-fbq74\"" Sep 08 17:09:42.574154 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.574129 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:09:42.599438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.599408 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c695l\" (UniqueName: \"kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.599600 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.599511 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.700885 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.700845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.701061 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.700955 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c695l\" (UniqueName: \"kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.701715 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.701681 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.722760 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.722736 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c695l\" (UniqueName: \"kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l\") pod \"payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:42.868204 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:42.868161 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:43.142807 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:43.142720 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:09:43.145770 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:09:43.145743 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcef052_dcdc_4fac_a30c_c49874ad49d5.slice/crio-93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518 WatchSource:0}: Error finding container 93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518: Status 404 returned error can't find the container with id 93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518 Sep 08 17:09:43.215895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:43.215864 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" event={"ID":"edcef052-dcdc-4fac-a30c-c49874ad49d5","Type":"ContainerStarted","Data":"93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518"} Sep 08 17:09:44.223126 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:44.223090 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" event={"ID":"edcef052-dcdc-4fac-a30c-c49874ad49d5","Type":"ContainerStarted","Data":"93c13aa702feb77e2e0076659caa08f226319aa5fa262219fa7e5bc488ee40dc"} Sep 08 17:09:44.223561 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:44.223196 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:09:44.243051 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:44.243000 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" podStartSLOduration=2.242983967 podStartE2EDuration="2.242983967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:44.23989056 +0000 UTC m=+1115.384661119" watchObservedRunningTime="2026-09-08 17:09:44.242983967 +0000 UTC m=+1115.387754550" Sep 08 17:09:50.506100 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:50.506045 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:09:55.230952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:09:55.230920 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:10:00.318811 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.318716 2556 generic.go:356] "Generic (PLEG): container finished" podID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerID="a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c" exitCode=0 Sep 08 17:10:00.318811 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.318785 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerDied","Data":"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c"} Sep 08 17:10:00.319228 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.319204 2556 scope.go:117] "RemoveContainer" containerID="a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c" Sep 08 17:10:00.320318 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.320291 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:10:00.320412 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.320393 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:10:00.320451 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:00.320426 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:10:01.326630 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:01.326589 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerStarted","Data":"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b"} Sep 08 17:10:01.347639 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:01.347603 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:10:01.347952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:01.347919 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:01.348044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:01.347979 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:02.347311 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:02.347266 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:02.347856 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:02.347365 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:03.349882 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:03.349836 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:03.350373 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:03.349902 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:04.347846 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:04.347803 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:04.348058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:04.347878 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:05.347394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:05.347358 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:05.347990 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:05.347960 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:06.347975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:06.347924 2556 patch_prober.go:28] interesting pod/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 17:10:06.348506 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:06.347999 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 17:10:06.907401 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:06.907368 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:10:07.351376 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:07.351275 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:10:07.351735 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:07.351540 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:10:07.352297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:07.352276 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:10:09.462857 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:09.462829 2556 scope.go:117] "RemoveContainer" containerID="a5d3b70d1affa859f2fc62d3e98480d4c7303dfa1f6bc2a4e01fa21ead11390a" Sep 08 17:10:28.412677 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:28.412640 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:10:42.004695 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:42.004610 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:10:49.825287 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.825252 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd"] Sep 08 17:10:49.828910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.828892 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:49.831390 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.831372 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:10:49.831452 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.831426 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:10:49.836843 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.836728 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd"] Sep 08 17:10:49.929926 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:49.929891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd\" (UID: \"33a43b2a-45ca-4c13-b502-70df0ef3241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:50.030629 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:50.030590 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd\" (UID: \"33a43b2a-45ca-4c13-b502-70df0ef3241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:50.031227 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:50.031206 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd\" (UID: \"33a43b2a-45ca-4c13-b502-70df0ef3241f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:50.140407 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:50.140377 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:50.410253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:50.410229 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd"] Sep 08 17:10:50.411874 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:10:50.411849 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a43b2a_45ca_4c13_b502_70df0ef3241f.slice/crio-ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242 WatchSource:0}: Error finding container ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242: Status 404 returned error can't find the container with id ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242 Sep 08 17:10:50.618959 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:50.618923 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" event={"ID":"33a43b2a-45ca-4c13-b502-70df0ef3241f","Type":"ContainerStarted","Data":"ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242"} Sep 08 17:10:51.626962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:51.626926 2556 generic.go:356] "Generic (PLEG): container finished" podID="33a43b2a-45ca-4c13-b502-70df0ef3241f" containerID="ec9a5a8cbb0dca95bc2784e24a9d80ebefd3155a0d86d5d88d8e0bddc47cf75e" exitCode=0 Sep 08 17:10:51.627370 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:51.627012 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" event={"ID":"33a43b2a-45ca-4c13-b502-70df0ef3241f","Type":"ContainerDied","Data":"ec9a5a8cbb0dca95bc2784e24a9d80ebefd3155a0d86d5d88d8e0bddc47cf75e"} Sep 08 17:10:52.770113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:52.770090 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:52.859696 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:52.859658 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca\") pod \"33a43b2a-45ca-4c13-b502-70df0ef3241f\" (UID: \"33a43b2a-45ca-4c13-b502-70df0ef3241f\") " Sep 08 17:10:52.860030 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:52.860000 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33a43b2a-45ca-4c13-b502-70df0ef3241f" (UID: "33a43b2a-45ca-4c13-b502-70df0ef3241f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:10:52.961624 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:52.961541 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a43b2a-45ca-4c13-b502-70df0ef3241f-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:10:53.639992 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:53.639955 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" event={"ID":"33a43b2a-45ca-4c13-b502-70df0ef3241f","Type":"ContainerDied","Data":"ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242"} Sep 08 17:10:53.639992 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:53.639990 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd" Sep 08 17:10:53.640207 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:10:53.640014 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea594ba2adc41090bb4e1eed48518b26270961326f3ba8f9ee46d285e8e0e242" Sep 08 17:11:00.805425 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:00.805387 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:11:05.475449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.475304 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:11:05.476407 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.476372 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" podUID="edcef052-dcdc-4fac-a30c-c49874ad49d5" containerName="payload-pre-processing" containerID="cri-o://93c13aa702feb77e2e0076659caa08f226319aa5fa262219fa7e5bc488ee40dc" gracePeriod=30 Sep 08 17:11:05.647456 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.647427 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd"] Sep 08 17:11:05.649750 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.649724 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb1565be-wbbfd"] Sep 08 17:11:05.735406 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.734685 2556 generic.go:356] "Generic (PLEG): container finished" podID="edcef052-dcdc-4fac-a30c-c49874ad49d5" containerID="93c13aa702feb77e2e0076659caa08f226319aa5fa262219fa7e5bc488ee40dc" exitCode=0 Sep 08 17:11:05.735406 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.734744 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" event={"ID":"edcef052-dcdc-4fac-a30c-c49874ad49d5","Type":"ContainerDied","Data":"93c13aa702feb77e2e0076659caa08f226319aa5fa262219fa7e5bc488ee40dc"} Sep 08 17:11:05.735406 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.734794 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" event={"ID":"edcef052-dcdc-4fac-a30c-c49874ad49d5","Type":"ContainerDied","Data":"93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518"} Sep 08 17:11:05.735406 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.734818 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b369dcfa7f5dd8ca30c449321526f7ec01fde4e3a9ff145d5f3ebfe0df1518" Sep 08 17:11:05.755744 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.755655 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:11:05.795836 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.795805 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c695l\" (UniqueName: \"kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l\") pod \"edcef052-dcdc-4fac-a30c-c49874ad49d5\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " Sep 08 17:11:05.796008 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.795933 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume\") pod \"edcef052-dcdc-4fac-a30c-c49874ad49d5\" (UID: \"edcef052-dcdc-4fac-a30c-c49874ad49d5\") " Sep 08 17:11:05.796298 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.796265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edcef052-dcdc-4fac-a30c-c49874ad49d5" (UID: "edcef052-dcdc-4fac-a30c-c49874ad49d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:11:05.798194 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.798161 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l" (OuterVolumeSpecName: "kube-api-access-c695l") pod "edcef052-dcdc-4fac-a30c-c49874ad49d5" (UID: "edcef052-dcdc-4fac-a30c-c49874ad49d5"). InnerVolumeSpecName "kube-api-access-c695l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:11:05.897801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.897767 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c695l\" (UniqueName: \"kubernetes.io/projected/edcef052-dcdc-4fac-a30c-c49874ad49d5-kube-api-access-c695l\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:11:05.897801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:05.897795 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcef052-dcdc-4fac-a30c-c49874ad49d5-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:11:06.740625 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:06.740594 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z" Sep 08 17:11:06.764124 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:06.764096 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:11:06.767868 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:06.767844 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb1565be-674c7b4f7d-gqb2z"] Sep 08 17:11:07.343176 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:07.343140 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a43b2a-45ca-4c13-b502-70df0ef3241f" path="/var/lib/kubelet/pods/33a43b2a-45ca-4c13-b502-70df0ef3241f/volumes" Sep 08 17:11:07.343560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:07.343546 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcef052-dcdc-4fac-a30c-c49874ad49d5" path="/var/lib/kubelet/pods/edcef052-dcdc-4fac-a30c-c49874ad49d5/volumes" Sep 08 17:11:09.487968 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:09.487937 2556 scope.go:117] "RemoveContainer" containerID="d5d7158d36e88cb267bf2fc039a7a122ede6efd0f88125e8abfa1f240b83370b" Sep 08 17:11:17.216068 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:17.216031 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:11:33.007542 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:33.007501 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:11:48.008264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:48.008227 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:11:56.513916 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:56.513880 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:11:56.514468 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:11:56.514125 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" containerID="cri-o://3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b" gracePeriod=30 Sep 08 17:12:01.770731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.770707 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:12:01.913178 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913143 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mxs8\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913190 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913232 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913254 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913298 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913376 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913404 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913403 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913454 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913496 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy\") pod \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\" (UID: \"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c\") " Sep 08 17:12:01.913797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913553 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:12:01.913797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913677 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:12:01.913797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913734 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:12:01.914045 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.913976 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:12:01.914045 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.914002 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:01.914045 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.914023 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:01.914045 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.914038 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:01.914204 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.914042 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:12:01.915705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.915679 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:12:01.915705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.915694 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:12:01.915905 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.915887 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:12:01.915967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:01.915914 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8" (OuterVolumeSpecName: "kube-api-access-8mxs8") pod "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" (UID: "f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c"). InnerVolumeSpecName "kube-api-access-8mxs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:12:02.015590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015554 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxs8\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-kube-api-access-8mxs8\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.015590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015584 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.015590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015596 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.015831 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015605 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.015831 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015613 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.015831 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.015621 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:12:02.048767 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.048736 2556 generic.go:356] "Generic (PLEG): container finished" podID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerID="3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b" exitCode=0 Sep 08 17:12:02.048996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.048781 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerDied","Data":"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b"} Sep 08 17:12:02.048996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.048786 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" Sep 08 17:12:02.048996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.048823 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4" event={"ID":"f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c","Type":"ContainerDied","Data":"0bedea3c3ebecd8fb4b5506ac321a5da7cdf0c394385d5d209e287011e2a8326"} Sep 08 17:12:02.048996 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.048845 2556 scope.go:117] "RemoveContainer" containerID="3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b" Sep 08 17:12:02.058623 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.058602 2556 scope.go:117] "RemoveContainer" containerID="a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c" Sep 08 17:12:02.067452 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.067433 2556 scope.go:117] "RemoveContainer" containerID="3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b" Sep 08 17:12:02.067724 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:12:02.067704 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b\": container with ID starting with 3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b not found: ID does not exist" containerID="3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b" Sep 08 17:12:02.067766 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.067733 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b"} err="failed to get container status \"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b\": rpc error: code = NotFound desc = could not find container \"3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b\": container with ID starting with 3ad29accc67347774b26bdd30e17791b8e38e015288efd35b191e3bf2b71017b not found: ID does not exist" Sep 08 17:12:02.067766 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.067751 2556 scope.go:117] "RemoveContainer" containerID="a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c" Sep 08 17:12:02.068001 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:12:02.067984 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c\": container with ID starting with a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c not found: ID does not exist" containerID="a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c" Sep 08 17:12:02.068054 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.068007 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c"} err="failed to get container status \"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c\": rpc error: code = NotFound desc = could not find container \"a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c\": container with ID starting with a01f7bb60286ad8c16f4dcca06e6727a64424dc5220ba15fc0cc1202c7ad5a0c not found: ID does not exist" Sep 08 17:12:02.071772 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.071749 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:12:02.075992 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:02.075972 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb1565be-openshift-default-65f86c5b7b-42st4"] Sep 08 17:12:03.343397 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:03.343363 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" path="/var/lib/kubelet/pods/f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c/volumes" Sep 08 17:12:05.303539 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:05.303505 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:12:09.508908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:09.508874 2556 scope.go:117] "RemoveContainer" containerID="5e1b900b37e7c3b0676414f223797653a3b3a98bb60f9d1df2c0da33fd1ace17" Sep 08 17:12:21.503440 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:21.503405 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:12:38.202916 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:38.202883 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:12:50.639561 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.639528 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:12:50.639975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.639961 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a43b2a-45ca-4c13-b502-70df0ef3241f" containerName="revoke-keys" Sep 08 17:12:50.639975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.639975 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a43b2a-45ca-4c13-b502-70df0ef3241f" containerName="revoke-keys" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.639996 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640001 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640008 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640014 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640028 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcef052-dcdc-4fac-a30c-c49874ad49d5" containerName="payload-pre-processing" Sep 08 17:12:50.640058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640033 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcef052-dcdc-4fac-a30c-c49874ad49d5" containerName="payload-pre-processing" Sep 08 17:12:50.640240 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640109 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:12:50.640240 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640118 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcef052-dcdc-4fac-a30c-c49874ad49d5" containerName="payload-pre-processing" Sep 08 17:12:50.640240 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.640125 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a43b2a-45ca-4c13-b502-70df0ef3241f" containerName="revoke-keys" Sep 08 17:12:50.643287 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.643267 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.646610 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.646588 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f0d25d-openshift-default-dockercfg-5w64p\"" Sep 08 17:12:50.658675 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.658647 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:12:50.712155 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712120 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712364 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712169 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712364 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712206 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712364 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712412 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2zm\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712438 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712511 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712467 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.712661 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.712520 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813775 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813807 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813839 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813868 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.813940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814226 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.813973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814226 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2zm\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814226 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814382 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814499 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814445 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814556 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814517 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814619 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814567 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.814748 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.814727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.816263 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.816236 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.816407 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.816397 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.834500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.834473 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2zm\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.845773 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.845742 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token\") pod \"e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:50.953802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:50.953714 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:51.234571 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.234545 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:12:51.236450 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:12:51.236422 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350428cf_c76c_4870_a1f5_d6b68fcd2275.slice/crio-87d5744a33da2b337baf3cd6e1a26a4661f57705fbd6bb3bfe7cd13ea6d1bb58 WatchSource:0}: Error finding container 87d5744a33da2b337baf3cd6e1a26a4661f57705fbd6bb3bfe7cd13ea6d1bb58: Status 404 returned error can't find the container with id 87d5744a33da2b337baf3cd6e1a26a4661f57705fbd6bb3bfe7cd13ea6d1bb58 Sep 08 17:12:51.238627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.238595 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:12:51.238700 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.238657 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:12:51.238700 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.238685 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:12:51.324677 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.324636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerStarted","Data":"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480"} Sep 08 17:12:51.324880 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.324712 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerStarted","Data":"87d5744a33da2b337baf3cd6e1a26a4661f57705fbd6bb3bfe7cd13ea6d1bb58"} Sep 08 17:12:51.345536 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.345483 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podStartSLOduration=1.345466575 podStartE2EDuration="1.345466575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:51.345121679 +0000 UTC m=+1302.489892241" watchObservedRunningTime="2026-09-08 17:12:51.345466575 +0000 UTC m=+1302.490237136" Sep 08 17:12:51.954007 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:51.953972 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:52.959661 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:52.959630 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:53.335538 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:53.335455 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:12:53.343088 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:12:53.343061 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:13:02.705374 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:02.705336 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:13:09.532947 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:09.532917 2556 scope.go:117] "RemoveContainer" containerID="2171306cc5f290c647015d58c6fcbc322290f5bbc577f30828f3a93a2d0dca71" Sep 08 17:13:13.446561 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.446528 2556 generic.go:356] "Generic (PLEG): container finished" podID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerID="879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480" exitCode=0 Sep 08 17:13:13.446993 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.446599 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerDied","Data":"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480"} Sep 08 17:13:13.447042 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.447009 2556 scope.go:117] "RemoveContainer" containerID="879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480" Sep 08 17:13:13.448200 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.448175 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:13.448268 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.448237 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:13.448304 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:13.448268 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:14.458188 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:14.458147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerStarted","Data":"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335"} Sep 08 17:13:14.954743 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:14.954694 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:13:14.955125 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:14.954902 2556 patch_prober.go:28] interesting pod/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 08 17:13:14.955346 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:14.955284 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 08 17:13:15.955032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:15.954991 2556 patch_prober.go:28] interesting pod/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 08 17:13:15.955665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:15.955631 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 08 17:13:16.955071 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:16.955032 2556 patch_prober.go:28] interesting pod/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 08 17:13:16.955717 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:16.955684 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 08 17:13:17.954164 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:17.954116 2556 patch_prober.go:28] interesting pod/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 08 17:13:17.954424 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:17.954187 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 08 17:13:18.960147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.960114 2556 patch_prober.go:28] interesting pod/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 08 17:13:18.960808 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.960765 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 08 17:13:18.962246 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.962225 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:13:18.973796 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.973771 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52f3ef4-ac74-4369-8a5c-2c6f8f25df3c" containerName="istio-proxy" Sep 08 17:13:18.978203 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.978182 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:18.981161 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.981132 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b9f2b8-openshift-default-dockercfg-kgbsp\"" Sep 08 17:13:18.995465 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:18.995437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:13:19.031283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031245 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031287 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031551 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031448 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031551 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031551 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031535 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031571 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031603 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7nb\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031708 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.031848 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.031730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132575 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132627 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132661 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7nb\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.132771 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133057 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132813 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133057 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132869 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133057 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.132899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133422 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.133398 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133683 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.133648 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133774 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.133659 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.133948 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.133926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.134208 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.134188 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.135643 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.135624 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.135843 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.135827 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.144980 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.144949 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7nb\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.146778 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.146750 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token\") pod \"e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.289563 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.289470 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:19.566072 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.566049 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:13:19.568807 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:13:19.568771 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f35a91_c545_43da_9f63_9a509eb26602.slice/crio-4607bfadd51ac9a9f510c026d5bcc33947e8df0a7415cf17caddad0c0a425d4a WatchSource:0}: Error finding container 4607bfadd51ac9a9f510c026d5bcc33947e8df0a7415cf17caddad0c0a425d4a: Status 404 returned error can't find the container with id 4607bfadd51ac9a9f510c026d5bcc33947e8df0a7415cf17caddad0c0a425d4a Sep 08 17:13:19.571260 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.571226 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:19.571373 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.571295 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:19.571373 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.571353 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:19.958530 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.958499 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:13:19.958919 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.958895 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:13:19.959596 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:19.959579 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:13:20.497405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:20.497369 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerStarted","Data":"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5"} Sep 08 17:13:20.497838 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:20.497424 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerStarted","Data":"4607bfadd51ac9a9f510c026d5bcc33947e8df0a7415cf17caddad0c0a425d4a"} Sep 08 17:13:20.517681 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:20.517632 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podStartSLOduration=2.517615664 podStartE2EDuration="2.517615664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:13:20.5166819 +0000 UTC m=+1331.661452473" watchObservedRunningTime="2026-09-08 17:13:20.517615664 +0000 UTC m=+1331.662386225" Sep 08 17:13:21.290607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:21.290567 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:21.296253 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:21.296218 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:21.503436 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:21.503403 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:21.504185 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:21.504161 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:24.362850 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.362814 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:13:24.366790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.366768 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.369565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.369544 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b9f2b8-dockercfg-xmqhc\"" Sep 08 17:13:24.369694 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.369586 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b9f2b8\"" Sep 08 17:13:24.381117 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.381088 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:13:24.493356 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.493299 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.494917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.494886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxx8s\" (UniqueName: \"kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.595974 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.595936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.596166 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.596003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxx8s\" (UniqueName: \"kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.596766 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.596743 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.605509 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.605481 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxx8s\" (UniqueName: \"kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s\") pod \"payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.677981 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.677945 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:24.952654 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:24.952622 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:13:24.954801 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:13:24.954775 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb23af7_62f0_49ef_ba0d_2af9e46b4f50.slice/crio-39d9338888bb0fb291fd17e8f3c0d7d803c4c322350c27c7cec50c1c3ae09234 WatchSource:0}: Error finding container 39d9338888bb0fb291fd17e8f3c0d7d803c4c322350c27c7cec50c1c3ae09234: Status 404 returned error can't find the container with id 39d9338888bb0fb291fd17e8f3c0d7d803c4c322350c27c7cec50c1c3ae09234 Sep 08 17:13:25.526472 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:25.526436 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" event={"ID":"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50","Type":"ContainerStarted","Data":"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da"} Sep 08 17:13:25.526879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:25.526496 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" event={"ID":"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50","Type":"ContainerStarted","Data":"39d9338888bb0fb291fd17e8f3c0d7d803c4c322350c27c7cec50c1c3ae09234"} Sep 08 17:13:25.526879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:25.526526 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:25.544667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:25.544618 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" podStartSLOduration=1.544599743 podStartE2EDuration="1.544599743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:13:25.543862306 +0000 UTC m=+1336.688632873" watchObservedRunningTime="2026-09-08 17:13:25.544599743 +0000 UTC m=+1336.689370307" Sep 08 17:13:34.008399 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:34.008308 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:13:36.533349 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:36.533306 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:13:44.646460 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.646422 2556 generic.go:356] "Generic (PLEG): container finished" podID="95f35a91-c545-43da-9f63-9a509eb26602" containerID="4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5" exitCode=0 Sep 08 17:13:44.646882 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.646493 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerDied","Data":"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5"} Sep 08 17:13:44.646882 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.646869 2556 scope.go:117] "RemoveContainer" containerID="4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5" Sep 08 17:13:44.647977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.647947 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:44.648041 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.648018 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:44.648077 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:44.648052 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:13:45.659070 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:45.658060 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerStarted","Data":"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0"} Sep 08 17:13:46.290303 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:46.290265 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:46.290562 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:46.290539 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 08 17:13:46.290610 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:46.290589 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 08 17:13:47.289873 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:47.289835 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 08 17:13:47.290277 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:47.289897 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 08 17:13:48.290282 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:48.290243 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 08 17:13:48.290689 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:48.290309 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 08 17:13:49.290450 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:49.290412 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 08 17:13:49.290850 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:49.290472 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 08 17:13:50.236157 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.236125 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc"] Sep 08 17:13:50.239950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.239934 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:50.242622 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.242596 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:13:50.242799 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.242595 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:13:50.255114 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.255083 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc"] Sep 08 17:13:50.284023 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.283988 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc\" (UID: \"d33559a8-4ac9-422e-928a-02a034c1c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:50.290007 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.289973 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 08 17:13:50.290143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.290040 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 08 17:13:50.385614 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.385580 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc\" (UID: \"d33559a8-4ac9-422e-928a-02a034c1c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:50.386201 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.386183 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc\" (UID: \"d33559a8-4ac9-422e-928a-02a034c1c0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:50.571948 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.571857 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:50.848939 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:50.848915 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc"] Sep 08 17:13:50.851334 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:13:50.851294 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33559a8_4ac9_422e_928a_02a034c1c0a6.slice/crio-6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b WatchSource:0}: Error finding container 6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b: Status 404 returned error can't find the container with id 6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b Sep 08 17:13:51.294477 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.294447 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:51.294742 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.294720 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:51.295466 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.295444 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:13:51.696606 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.696565 2556 generic.go:356] "Generic (PLEG): container finished" podID="d33559a8-4ac9-422e-928a-02a034c1c0a6" containerID="588ca5b78946548f1992edb8970fd7c199f04f84b367ba48c88cd121b6809b1d" exitCode=0 Sep 08 17:13:51.697027 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.696660 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" event={"ID":"d33559a8-4ac9-422e-928a-02a034c1c0a6","Type":"ContainerDied","Data":"588ca5b78946548f1992edb8970fd7c199f04f84b367ba48c88cd121b6809b1d"} Sep 08 17:13:51.697027 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:51.696727 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" event={"ID":"d33559a8-4ac9-422e-928a-02a034c1c0a6","Type":"ContainerStarted","Data":"6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b"} Sep 08 17:13:52.840400 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:52.840377 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:52.913225 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:52.913182 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca\") pod \"d33559a8-4ac9-422e-928a-02a034c1c0a6\" (UID: \"d33559a8-4ac9-422e-928a-02a034c1c0a6\") " Sep 08 17:13:52.913571 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:52.913546 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d33559a8-4ac9-422e-928a-02a034c1c0a6" (UID: "d33559a8-4ac9-422e-928a-02a034c1c0a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:13:53.014193 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:53.014107 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d33559a8-4ac9-422e-928a-02a034c1c0a6-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:13:53.710301 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:53.710267 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" event={"ID":"d33559a8-4ac9-422e-928a-02a034c1c0a6","Type":"ContainerDied","Data":"6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b"} Sep 08 17:13:53.710301 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:53.710297 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc" Sep 08 17:13:53.710790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:53.710317 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea9d958a213cf4e393ec310a463838ae81b9b1d20ef5da24d2c80e7a5bbce0b" Sep 08 17:13:59.711242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:13:59.711205 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:14:06.404764 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:06.404725 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc"] Sep 08 17:14:06.406883 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:06.406857 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f0d25d-vzdkc"] Sep 08 17:14:07.343393 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:07.343356 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d33559a8-4ac9-422e-928a-02a034c1c0a6" path="/var/lib/kubelet/pods/d33559a8-4ac9-422e-928a-02a034c1c0a6/volumes" Sep 08 17:14:09.559106 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:09.559075 2556 scope.go:117] "RemoveContainer" containerID="88eb82088834ab688e57cd1e52df2381a4aa2f6ba760abe5d44a9e6fa42af9ee" Sep 08 17:14:09.571012 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:09.570990 2556 scope.go:117] "RemoveContainer" containerID="96e43b3e3bd107849cce440449517f270af457385fb8674679201e4879c79584" Sep 08 17:14:12.972797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:12.972754 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:14:12.973273 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:12.973043 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" containerID="cri-o://0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335" gracePeriod=30 Sep 08 17:14:18.232230 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.232206 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:14:18.282033 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.281947 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282033 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.281984 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282033 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282019 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282068 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282091 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282183 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp2zm\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282212 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282280 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:18.282315 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282302 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282369 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:18.282665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282383 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy\") pod \"350428cf-c76c-4870-a1f5-d6b68fcd2275\" (UID: \"350428cf-c76c-4870-a1f5-d6b68fcd2275\") " Sep 08 17:14:18.282665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282386 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:18.282665 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282464 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:18.282876 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282731 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data" (OuterVolumeSpecName: "istio-data") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:18.282953 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282928 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.283009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282954 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/350428cf-c76c-4870-a1f5-d6b68fcd2275-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.283009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282968 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.283009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282981 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.283009 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.282994 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.284536 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.284513 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:14:18.284956 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.284937 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm" (OuterVolumeSpecName: "kube-api-access-mp2zm") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "kube-api-access-mp2zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:18.285025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.284958 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token" (OuterVolumeSpecName: "istio-token") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:18.285025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.284966 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "350428cf-c76c-4870-a1f5-d6b68fcd2275" (UID: "350428cf-c76c-4870-a1f5-d6b68fcd2275"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:18.384234 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.384203 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.384234 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.384228 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.384234 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.384236 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.384476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.384244 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2zm\" (UniqueName: \"kubernetes.io/projected/350428cf-c76c-4870-a1f5-d6b68fcd2275-kube-api-access-mp2zm\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:18.853448 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.853416 2556 generic.go:356] "Generic (PLEG): container finished" podID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerID="0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335" exitCode=0 Sep 08 17:14:18.853663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.853470 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" Sep 08 17:14:18.853663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.853497 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerDied","Data":"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335"} Sep 08 17:14:18.853663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.853551 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9" event={"ID":"350428cf-c76c-4870-a1f5-d6b68fcd2275","Type":"ContainerDied","Data":"87d5744a33da2b337baf3cd6e1a26a4661f57705fbd6bb3bfe7cd13ea6d1bb58"} Sep 08 17:14:18.853663 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.853574 2556 scope.go:117] "RemoveContainer" containerID="0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335" Sep 08 17:14:18.871335 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.871301 2556 scope.go:117] "RemoveContainer" containerID="879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480" Sep 08 17:14:18.880166 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.880144 2556 scope.go:117] "RemoveContainer" containerID="0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335" Sep 08 17:14:18.880465 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.880445 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:14:18.880465 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:18.880452 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335\": container with ID starting with 0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335 not found: ID does not exist" containerID="0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335" Sep 08 17:14:18.880556 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.880482 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335"} err="failed to get container status \"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335\": rpc error: code = NotFound desc = could not find container \"0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335\": container with ID starting with 0c218fc6fe1830cb17d034e5603a29f1883778f8f3cc72c2ec6ca843d8027335 not found: ID does not exist" Sep 08 17:14:18.880556 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.880502 2556 scope.go:117] "RemoveContainer" containerID="879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480" Sep 08 17:14:18.880807 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:18.880790 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480\": container with ID starting with 879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480 not found: ID does not exist" containerID="879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480" Sep 08 17:14:18.880858 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.880809 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480"} err="failed to get container status \"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480\": rpc error: code = NotFound desc = could not find container \"879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480\": container with ID starting with 879f230a9e4b5da4149b526154041bceae234bd118bd54df903d19e6297f4480 not found: ID does not exist" Sep 08 17:14:18.884311 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:18.884290 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f0d25d-openshift-default-58d6f55646-gsps9"] Sep 08 17:14:19.342627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:19.342596 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" path="/var/lib/kubelet/pods/350428cf-c76c-4870-a1f5-d6b68fcd2275/volumes" Sep 08 17:14:22.798228 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:22.798185 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:14:29.167791 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.167754 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:14:29.168273 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.168000 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" podUID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" containerName="payload-pre-processing" containerID="cri-o://209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da" gracePeriod=30 Sep 08 17:14:29.425723 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.425660 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:14:29.504370 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.503642 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxx8s\" (UniqueName: \"kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s\") pod \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " Sep 08 17:14:29.504370 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.503715 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume\") pod \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\" (UID: \"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50\") " Sep 08 17:14:29.505287 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.504983 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" (UID: "0cb23af7-62f0-49ef-ba0d-2af9e46b4f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:29.514956 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.514898 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s" (OuterVolumeSpecName: "kube-api-access-pxx8s") pod "0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" (UID: "0cb23af7-62f0-49ef-ba0d-2af9e46b4f50"). InnerVolumeSpecName "kube-api-access-pxx8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:29.604860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.604820 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxx8s\" (UniqueName: \"kubernetes.io/projected/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-kube-api-access-pxx8s\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.604860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.604859 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:29.914335 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.914286 2556 generic.go:356] "Generic (PLEG): container finished" podID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" containerID="209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da" exitCode=0 Sep 08 17:14:29.914519 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.914339 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" Sep 08 17:14:29.914519 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.914373 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" event={"ID":"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50","Type":"ContainerDied","Data":"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da"} Sep 08 17:14:29.914519 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.914428 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8" event={"ID":"0cb23af7-62f0-49ef-ba0d-2af9e46b4f50","Type":"ContainerDied","Data":"39d9338888bb0fb291fd17e8f3c0d7d803c4c322350c27c7cec50c1c3ae09234"} Sep 08 17:14:29.914519 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.914458 2556 scope.go:117] "RemoveContainer" containerID="209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da" Sep 08 17:14:29.931863 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.931844 2556 scope.go:117] "RemoveContainer" containerID="209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da" Sep 08 17:14:29.932172 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:29.932149 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da\": container with ID starting with 209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da not found: ID does not exist" containerID="209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da" Sep 08 17:14:29.932231 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.932181 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da"} err="failed to get container status \"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da\": rpc error: code = NotFound desc = could not find container \"209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da\": container with ID starting with 209027eb90fb3c34768945d365900d3af16fad6e045d9e243ccd95d781b9f4da not found: ID does not exist" Sep 08 17:14:29.942370 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.942345 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:14:29.946739 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:29.946717 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9f2b8-7c64445878-vxrx8"] Sep 08 17:14:31.343242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:31.343205 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" path="/var/lib/kubelet/pods/0cb23af7-62f0-49ef-ba0d-2af9e46b4f50/volumes" Sep 08 17:14:36.124514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:36.124475 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:14:36.125011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:36.124780 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" containerID="cri-o://84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0" gracePeriod=30 Sep 08 17:14:36.296786 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:36.296748 2556 patch_prober.go:28] interesting pod/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:14:36.296959 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:36.296830 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:14:41.387108 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.387041 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:14:41.524507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524468 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524507 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524509 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524588 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524615 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524659 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524687 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524721 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524716 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524746 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.524770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524773 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs7nb\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb\") pod \"95f35a91-c545-43da-9f63-9a509eb26602\" (UID: \"95f35a91-c545-43da-9f63-9a509eb26602\") " Sep 08 17:14:41.525189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.524934 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:41.525189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525002 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:41.525189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525022 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data" (OuterVolumeSpecName: "istio-data") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:41.525189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525085 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:41.525418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525212 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.525418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525232 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.525418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525247 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.525418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525260 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f35a91-c545-43da-9f63-9a509eb26602-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.525418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.525272 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.526927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.526905 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token" (OuterVolumeSpecName: "istio-token") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:41.527037 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.526933 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:14:41.527272 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.527256 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:41.527272 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.527265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb" (OuterVolumeSpecName: "kube-api-access-zs7nb") pod "95f35a91-c545-43da-9f63-9a509eb26602" (UID: "95f35a91-c545-43da-9f63-9a509eb26602"). InnerVolumeSpecName "kube-api-access-zs7nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:41.626834 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.626780 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f35a91-c545-43da-9f63-9a509eb26602-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.626834 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.626826 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.626834 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.626835 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f35a91-c545-43da-9f63-9a509eb26602-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.626834 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.626843 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7nb\" (UniqueName: \"kubernetes.io/projected/95f35a91-c545-43da-9f63-9a509eb26602-kube-api-access-zs7nb\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:41.981067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.981031 2556 generic.go:356] "Generic (PLEG): container finished" podID="95f35a91-c545-43da-9f63-9a509eb26602" containerID="84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0" exitCode=0 Sep 08 17:14:41.981249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.981079 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" Sep 08 17:14:41.981249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.981124 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerDied","Data":"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0"} Sep 08 17:14:41.981249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.981180 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs" event={"ID":"95f35a91-c545-43da-9f63-9a509eb26602","Type":"ContainerDied","Data":"4607bfadd51ac9a9f510c026d5bcc33947e8df0a7415cf17caddad0c0a425d4a"} Sep 08 17:14:41.981249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.981203 2556 scope.go:117] "RemoveContainer" containerID="84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0" Sep 08 17:14:41.990892 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.990874 2556 scope.go:117] "RemoveContainer" containerID="4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5" Sep 08 17:14:41.999184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.999164 2556 scope.go:117] "RemoveContainer" containerID="84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0" Sep 08 17:14:41.999503 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:41.999478 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0\": container with ID starting with 84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0 not found: ID does not exist" containerID="84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0" Sep 08 17:14:41.999602 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.999510 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0"} err="failed to get container status \"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0\": rpc error: code = NotFound desc = could not find container \"84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0\": container with ID starting with 84387e1041a4a2e6b49d62bcc5b790aadc51d2a3d5b318559a8ad2e4395417c0 not found: ID does not exist" Sep 08 17:14:41.999602 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.999528 2556 scope.go:117] "RemoveContainer" containerID="4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5" Sep 08 17:14:41.999751 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:41.999735 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5\": container with ID starting with 4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5 not found: ID does not exist" containerID="4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5" Sep 08 17:14:41.999791 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:41.999752 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5"} err="failed to get container status \"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5\": rpc error: code = NotFound desc = could not find container \"4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5\": container with ID starting with 4505e7b204989693410d4c1ae5df59c2e25ae8698e29598158d0fd619a64b3b5 not found: ID does not exist" Sep 08 17:14:42.003931 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:42.003903 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:14:42.008316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:42.008293 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9f2b8-openshift-default-65b46dbd7b-8rbfs"] Sep 08 17:14:42.398215 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:42.398176 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:14:43.342864 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:43.342829 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f35a91-c545-43da-9f63-9a509eb26602" path="/var/lib/kubelet/pods/95f35a91-c545-43da-9f63-9a509eb26602/volumes" Sep 08 17:14:46.990735 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:46.990701 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:14:46.991206 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:46.990954 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" podUID="2c9d803a-5338-482f-8d66-c774b82108c9" containerName="payload-processing" containerID="cri-o://fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363" gracePeriod=30 Sep 08 17:14:46.998576 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:46.998155 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:14:46.998754 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:46.998708 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" podUID="900beb2f-d926-4cad-b526-911caba07887" containerName="payload-pre-processing" containerID="cri-o://372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f" gracePeriod=30 Sep 08 17:14:47.248587 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.248521 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:14:47.283168 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.283141 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:14:47.387516 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.387483 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl5ft\" (UniqueName: \"kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft\") pod \"900beb2f-d926-4cad-b526-911caba07887\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " Sep 08 17:14:47.387696 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.387633 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume\") pod \"900beb2f-d926-4cad-b526-911caba07887\" (UID: \"900beb2f-d926-4cad-b526-911caba07887\") " Sep 08 17:14:47.387696 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.387659 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrkbd\" (UniqueName: \"kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd\") pod \"2c9d803a-5338-482f-8d66-c774b82108c9\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " Sep 08 17:14:47.387696 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.387683 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume\") pod \"2c9d803a-5338-482f-8d66-c774b82108c9\" (UID: \"2c9d803a-5338-482f-8d66-c774b82108c9\") " Sep 08 17:14:47.388053 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.388026 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "900beb2f-d926-4cad-b526-911caba07887" (UID: "900beb2f-d926-4cad-b526-911caba07887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:47.388138 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.388085 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c9d803a-5338-482f-8d66-c774b82108c9" (UID: "2c9d803a-5338-482f-8d66-c774b82108c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:47.389647 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.389623 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft" (OuterVolumeSpecName: "kube-api-access-nl5ft") pod "900beb2f-d926-4cad-b526-911caba07887" (UID: "900beb2f-d926-4cad-b526-911caba07887"). InnerVolumeSpecName "kube-api-access-nl5ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:47.389775 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.389750 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd" (OuterVolumeSpecName: "kube-api-access-wrkbd") pod "2c9d803a-5338-482f-8d66-c774b82108c9" (UID: "2c9d803a-5338-482f-8d66-c774b82108c9"). InnerVolumeSpecName "kube-api-access-wrkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:47.488724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.488692 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/900beb2f-d926-4cad-b526-911caba07887-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.488724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.488719 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkbd\" (UniqueName: \"kubernetes.io/projected/2c9d803a-5338-482f-8d66-c774b82108c9-kube-api-access-wrkbd\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.488724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.488728 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9d803a-5338-482f-8d66-c774b82108c9-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:47.488970 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:47.488746 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl5ft\" (UniqueName: \"kubernetes.io/projected/900beb2f-d926-4cad-b526-911caba07887-kube-api-access-nl5ft\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:48.017468 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.017434 2556 generic.go:356] "Generic (PLEG): container finished" podID="900beb2f-d926-4cad-b526-911caba07887" containerID="372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f" exitCode=0 Sep 08 17:14:48.017921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.017485 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" Sep 08 17:14:48.017921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.017529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" event={"ID":"900beb2f-d926-4cad-b526-911caba07887","Type":"ContainerDied","Data":"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f"} Sep 08 17:14:48.017921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.017600 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg" event={"ID":"900beb2f-d926-4cad-b526-911caba07887","Type":"ContainerDied","Data":"0cea311818e7f1d89eb9d2580d28d5c4b9b818227f08e48def052ccb46d2e853"} Sep 08 17:14:48.017921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.017635 2556 scope.go:117] "RemoveContainer" containerID="372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f" Sep 08 17:14:48.019082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.018918 2556 generic.go:356] "Generic (PLEG): container finished" podID="2c9d803a-5338-482f-8d66-c774b82108c9" containerID="fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363" exitCode=0 Sep 08 17:14:48.019082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.018975 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" Sep 08 17:14:48.019082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.019004 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" event={"ID":"2c9d803a-5338-482f-8d66-c774b82108c9","Type":"ContainerDied","Data":"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363"} Sep 08 17:14:48.019082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.019043 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642" event={"ID":"2c9d803a-5338-482f-8d66-c774b82108c9","Type":"ContainerDied","Data":"2178d29a7556b16bc7b2bae71f39545758856dddec298e56bc536e39151b6131"} Sep 08 17:14:48.026686 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.026666 2556 scope.go:117] "RemoveContainer" containerID="372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f" Sep 08 17:14:48.027020 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:48.026992 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f\": container with ID starting with 372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f not found: ID does not exist" containerID="372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f" Sep 08 17:14:48.027113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.027026 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f"} err="failed to get container status \"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f\": rpc error: code = NotFound desc = could not find container \"372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f\": container with ID starting with 372b94c2c0c1d0b0b3654284f38407c630f706cb9a533fa16bc7d9bcad965d3f not found: ID does not exist" Sep 08 17:14:48.027113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.027049 2556 scope.go:117] "RemoveContainer" containerID="fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363" Sep 08 17:14:48.037778 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.037758 2556 scope.go:117] "RemoveContainer" containerID="fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363" Sep 08 17:14:48.038035 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:48.038014 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363\": container with ID starting with fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363 not found: ID does not exist" containerID="fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363" Sep 08 17:14:48.038089 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.038041 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363"} err="failed to get container status \"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363\": rpc error: code = NotFound desc = could not find container \"fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363\": container with ID starting with fdab57a6792bd5d296a61ce572fca68844b7375883583ca86a2b00efc9c1a363 not found: ID does not exist" Sep 08 17:14:48.042368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.042341 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:14:48.046758 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.046733 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-ec96ff-6cf6947d94-ptcqg"] Sep 08 17:14:48.057256 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.057234 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:14:48.061571 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:48.061550 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec96ff-d8989c895-pf642"] Sep 08 17:14:49.343486 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:49.343455 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9d803a-5338-482f-8d66-c774b82108c9" path="/var/lib/kubelet/pods/2c9d803a-5338-482f-8d66-c774b82108c9/volumes" Sep 08 17:14:49.343932 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:49.343843 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="900beb2f-d926-4cad-b526-911caba07887" path="/var/lib/kubelet/pods/900beb2f-d926-4cad-b526-911caba07887/volumes" Sep 08 17:14:53.311372 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.311336 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:14:53.512607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.512566 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:14:53.512835 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.512789 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" containerID="cri-o://e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4" gracePeriod=30 Sep 08 17:14:53.859518 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859482 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt"] Sep 08 17:14:53.859881 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859869 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900beb2f-d926-4cad-b526-911caba07887" containerName="payload-pre-processing" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859882 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="900beb2f-d926-4cad-b526-911caba07887" containerName="payload-pre-processing" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859893 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9d803a-5338-482f-8d66-c774b82108c9" containerName="payload-processing" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859899 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9d803a-5338-482f-8d66-c774b82108c9" containerName="payload-processing" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859909 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859916 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859922 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33559a8-4ac9-422e-928a-02a034c1c0a6" containerName="revoke-keys" Sep 08 17:14:53.859927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859926 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33559a8-4ac9-422e-928a-02a034c1c0a6" containerName="revoke-keys" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859934 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859939 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859947 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859953 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859965 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859970 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859982 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" containerName="payload-pre-processing" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.859987 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" containerName="payload-pre-processing" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860060 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860067 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="350428cf-c76c-4870-a1f5-d6b68fcd2275" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860073 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="900beb2f-d926-4cad-b526-911caba07887" containerName="payload-pre-processing" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860079 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb23af7-62f0-49ef-ba0d-2af9e46b4f50" containerName="payload-pre-processing" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860088 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860094 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f35a91-c545-43da-9f63-9a509eb26602" containerName="istio-proxy" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860100 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d33559a8-4ac9-422e-928a-02a034c1c0a6" containerName="revoke-keys" Sep 08 17:14:53.860147 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.860106 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9d803a-5338-482f-8d66-c774b82108c9" containerName="payload-processing" Sep 08 17:14:53.864722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.864705 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:53.867932 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.867902 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:14:53.868080 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.868062 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:14:53.870853 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.870826 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt"] Sep 08 17:14:53.955242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:53.955208 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt\" (UID: \"62942868-bca4-4cab-aca8-07c8fdfcf2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:54.055921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:54.055885 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt\" (UID: \"62942868-bca4-4cab-aca8-07c8fdfcf2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:54.056518 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:54.056501 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt\" (UID: \"62942868-bca4-4cab-aca8-07c8fdfcf2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:54.177623 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:54.177574 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:54.447445 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:54.447418 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt"] Sep 08 17:14:54.449718 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:14:54.449694 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62942868_bca4_4cab_aca8_07c8fdfcf2bc.slice/crio-d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca WatchSource:0}: Error finding container d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca: Status 404 returned error can't find the container with id d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca Sep 08 17:14:55.059853 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:55.059820 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" event={"ID":"62942868-bca4-4cab-aca8-07c8fdfcf2bc","Type":"ContainerStarted","Data":"df2405907f9dd00cd9c2888075308f8fa7c79e45f7fa56fc2438c49bbb1f3977"} Sep 08 17:14:55.060032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:55.059881 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" event={"ID":"62942868-bca4-4cab-aca8-07c8fdfcf2bc","Type":"ContainerStarted","Data":"d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca"} Sep 08 17:14:55.075674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:55.075632 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" podStartSLOduration=2.075617494 podStartE2EDuration="2.075617494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:55.073727753 +0000 UTC m=+1426.218498314" watchObservedRunningTime="2026-09-08 17:14:55.075617494 +0000 UTC m=+1426.220388055" Sep 08 17:14:55.790626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:55.790582 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:14:55.791133 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:55.790668 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:14:56.066965 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:56.066882 2556 generic.go:356] "Generic (PLEG): container finished" podID="62942868-bca4-4cab-aca8-07c8fdfcf2bc" containerID="df2405907f9dd00cd9c2888075308f8fa7c79e45f7fa56fc2438c49bbb1f3977" exitCode=0 Sep 08 17:14:56.067113 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:56.066951 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" event={"ID":"62942868-bca4-4cab-aca8-07c8fdfcf2bc","Type":"ContainerDied","Data":"df2405907f9dd00cd9c2888075308f8fa7c79e45f7fa56fc2438c49bbb1f3977"} Sep 08 17:14:57.205947 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:57.205922 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:57.287967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:57.287931 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca\") pod \"62942868-bca4-4cab-aca8-07c8fdfcf2bc\" (UID: \"62942868-bca4-4cab-aca8-07c8fdfcf2bc\") " Sep 08 17:14:57.288348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:57.288302 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62942868-bca4-4cab-aca8-07c8fdfcf2bc" (UID: "62942868-bca4-4cab-aca8-07c8fdfcf2bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:57.389517 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:57.389483 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62942868-bca4-4cab-aca8-07c8fdfcf2bc-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.078110 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.078076 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" event={"ID":"62942868-bca4-4cab-aca8-07c8fdfcf2bc","Type":"ContainerDied","Data":"d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca"} Sep 08 17:14:58.078312 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.078128 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7dce420fc12b3f2aaf63c4d35cb76097bba0d18cb1101036923a701cffc7dca" Sep 08 17:14:58.078312 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.078081 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt" Sep 08 17:14:58.769800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.769777 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:14:58.805500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805421 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805462 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805500 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805502 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805518 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805658 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805679 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805707 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhx89\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.805770 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805721 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:58.806028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805763 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:58.806028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805776 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:58.806028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805752 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.806028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805846 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo\") pod \"2de5f12b-8360-4eed-98a2-a557d549bc05\" (UID: \"2de5f12b-8360-4eed-98a2-a557d549bc05\") " Sep 08 17:14:58.806028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.805973 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data" (OuterVolumeSpecName: "istio-data") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:58.806266 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806068 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:14:58.806353 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806298 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.806353 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806317 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2de5f12b-8360-4eed-98a2-a557d549bc05-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.806454 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806358 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.806454 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806372 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.806454 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.806388 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.807981 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.807959 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89" (OuterVolumeSpecName: "kube-api-access-nhx89") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "kube-api-access-nhx89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:58.808083 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.807971 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token" (OuterVolumeSpecName: "istio-token") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:14:58.808246 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.808217 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:14:58.808246 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.808230 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2de5f12b-8360-4eed-98a2-a557d549bc05" (UID: "2de5f12b-8360-4eed-98a2-a557d549bc05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:14:58.907712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.907668 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.907712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.907705 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhx89\" (UniqueName: \"kubernetes.io/projected/2de5f12b-8360-4eed-98a2-a557d549bc05-kube-api-access-nhx89\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.907712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.907715 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:58.907712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:58.907724 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2de5f12b-8360-4eed-98a2-a557d549bc05-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:14:59.086025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.085938 2556 generic.go:356] "Generic (PLEG): container finished" podID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerID="e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4" exitCode=0 Sep 08 17:14:59.086025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.085997 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerDied","Data":"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4"} Sep 08 17:14:59.086213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.086053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" event={"ID":"2de5f12b-8360-4eed-98a2-a557d549bc05","Type":"ContainerDied","Data":"7806333db68f3c134db606749b5fd25b40862a1b7f63a2aa62401e0f25d8abe5"} Sep 08 17:14:59.086213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.086006 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm" Sep 08 17:14:59.086213 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.086080 2556 scope.go:117] "RemoveContainer" containerID="e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4" Sep 08 17:14:59.104628 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.104605 2556 scope.go:117] "RemoveContainer" containerID="bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61" Sep 08 17:14:59.115441 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.115415 2556 scope.go:117] "RemoveContainer" containerID="e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4" Sep 08 17:14:59.115777 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:59.115752 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4\": container with ID starting with e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4 not found: ID does not exist" containerID="e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4" Sep 08 17:14:59.115855 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.115790 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4"} err="failed to get container status \"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4\": rpc error: code = NotFound desc = could not find container \"e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4\": container with ID starting with e274d38c76dfb211457c030c692fd052404c9264cb3ccfe38c6504d4862bf5e4 not found: ID does not exist" Sep 08 17:14:59.115855 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.115816 2556 scope.go:117] "RemoveContainer" containerID="bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61" Sep 08 17:14:59.116153 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:14:59.116090 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61\": container with ID starting with bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61 not found: ID does not exist" containerID="bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61" Sep 08 17:14:59.116153 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.116125 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61"} err="failed to get container status \"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61\": rpc error: code = NotFound desc = could not find container \"bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61\": container with ID starting with bf5736437a2f0de10d6f85f806fe6e6032b073ffebb74f0f5aae03cfba899a61 not found: ID does not exist" Sep 08 17:14:59.117611 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.117588 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:14:59.119638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.119618 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ec96ff-openshift-default-5c546b9d76-d5wjm"] Sep 08 17:14:59.343692 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:14:59.343615 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" path="/var/lib/kubelet/pods/2de5f12b-8360-4eed-98a2-a557d549bc05/volumes" Sep 08 17:15:00.136061 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136030 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26"] Sep 08 17:15:00.136461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136428 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.136461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136440 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.136461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136457 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.136461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136463 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.136599 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136474 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62942868-bca4-4cab-aca8-07c8fdfcf2bc" containerName="revoke-keys" Sep 08 17:15:00.136599 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136479 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="62942868-bca4-4cab-aca8-07c8fdfcf2bc" containerName="revoke-keys" Sep 08 17:15:00.136599 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136535 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.136599 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136543 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="62942868-bca4-4cab-aca8-07c8fdfcf2bc" containerName="revoke-keys" Sep 08 17:15:00.136599 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.136552 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de5f12b-8360-4eed-98a2-a557d549bc05" containerName="istio-proxy" Sep 08 17:15:00.144954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.141855 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48"] Sep 08 17:15:00.145574 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.145548 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" Sep 08 17:15:00.146626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.146606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" Sep 08 17:15:00.148635 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.148616 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:15:00.156928 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.155552 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26"] Sep 08 17:15:00.157712 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.157688 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48"] Sep 08 17:15:00.159896 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.159878 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" Sep 08 17:15:00.167701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.167681 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" Sep 08 17:15:00.441073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.440994 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26"] Sep 08 17:15:00.443468 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:15:00.443444 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f3ef02_3809_4f83_ae23_5930dc9aa7b7.slice/crio-16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68 WatchSource:0}: Error finding container 16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68: Status 404 returned error can't find the container with id 16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68 Sep 08 17:15:00.458032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:00.457945 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48"] Sep 08 17:15:01.099844 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.099731 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" event={"ID":"c0f3ef02-3809-4f83-ae23-5930dc9aa7b7","Type":"ContainerStarted","Data":"a2b3bef0a4f11df863a63d835d96cad464127dcc13ea1da2ebd9312f1d5e947f"} Sep 08 17:15:01.099844 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.099801 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" event={"ID":"c0f3ef02-3809-4f83-ae23-5930dc9aa7b7","Type":"ContainerStarted","Data":"16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68"} Sep 08 17:15:01.101184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.101162 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" event={"ID":"c5447325-2edf-4a2e-964e-5db88362534b","Type":"ContainerStarted","Data":"c57dac020985f75b96f2ffa19d700da7726971347101150de4863883743bc570"} Sep 08 17:15:01.101289 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.101206 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" event={"ID":"c5447325-2edf-4a2e-964e-5db88362534b","Type":"ContainerStarted","Data":"98cc63b0360c672039675f1041a7671e1726f9dad8a7227bc7bf75c32d97e2d7"} Sep 08 17:15:01.116790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.116739 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" podStartSLOduration=1.116719946 podStartE2EDuration="1.116719946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:15:01.115348011 +0000 UTC m=+1432.260118573" watchObservedRunningTime="2026-09-08 17:15:01.116719946 +0000 UTC m=+1432.261490507" Sep 08 17:15:01.130836 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:01.130791 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" podStartSLOduration=1.1307748420000001 podStartE2EDuration="1.130774842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:15:01.130189995 +0000 UTC m=+1432.274960557" watchObservedRunningTime="2026-09-08 17:15:01.130774842 +0000 UTC m=+1432.275545411" Sep 08 17:15:02.107514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:02.107430 2556 generic.go:356] "Generic (PLEG): container finished" podID="c0f3ef02-3809-4f83-ae23-5930dc9aa7b7" containerID="a2b3bef0a4f11df863a63d835d96cad464127dcc13ea1da2ebd9312f1d5e947f" exitCode=0 Sep 08 17:15:02.107908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:02.107523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" event={"ID":"c0f3ef02-3809-4f83-ae23-5930dc9aa7b7","Type":"ContainerDied","Data":"a2b3bef0a4f11df863a63d835d96cad464127dcc13ea1da2ebd9312f1d5e947f"} Sep 08 17:15:02.109098 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:02.109078 2556 generic.go:356] "Generic (PLEG): container finished" podID="c5447325-2edf-4a2e-964e-5db88362534b" containerID="c57dac020985f75b96f2ffa19d700da7726971347101150de4863883743bc570" exitCode=0 Sep 08 17:15:02.109197 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:02.109136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" event={"ID":"c5447325-2edf-4a2e-964e-5db88362534b","Type":"ContainerDied","Data":"c57dac020985f75b96f2ffa19d700da7726971347101150de4863883743bc570"} Sep 08 17:15:03.283923 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:03.283897 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" Sep 08 17:15:03.286819 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:03.286797 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" Sep 08 17:15:04.123134 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.123106 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" Sep 08 17:15:04.123356 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.123147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26" event={"ID":"c0f3ef02-3809-4f83-ae23-5930dc9aa7b7","Type":"ContainerDied","Data":"16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68"} Sep 08 17:15:04.123356 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.123196 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16af9edb1e6946df20743756ef0264f53e1c8a830072e4a741cd38af1d222b68" Sep 08 17:15:04.124631 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.124612 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" Sep 08 17:15:04.124777 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.124609 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48" event={"ID":"c5447325-2edf-4a2e-964e-5db88362534b","Type":"ContainerDied","Data":"98cc63b0360c672039675f1041a7671e1726f9dad8a7227bc7bf75c32d97e2d7"} Sep 08 17:15:04.124777 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.124750 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98cc63b0360c672039675f1041a7671e1726f9dad8a7227bc7bf75c32d97e2d7" Sep 08 17:15:04.823423 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.823370 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48"] Sep 08 17:15:04.826847 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:04.826816 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d7359b-29814795-92b48"] Sep 08 17:15:05.022900 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:05.022862 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt"] Sep 08 17:15:05.026128 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:05.026102 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d7359b-ztnzt"] Sep 08 17:15:05.342300 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:05.342253 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62942868-bca4-4cab-aca8-07c8fdfcf2bc" path="/var/lib/kubelet/pods/62942868-bca4-4cab-aca8-07c8fdfcf2bc/volumes" Sep 08 17:15:05.342631 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:05.342617 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5447325-2edf-4a2e-964e-5db88362534b" path="/var/lib/kubelet/pods/c5447325-2edf-4a2e-964e-5db88362534b/volumes" Sep 08 17:15:08.998239 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:08.998204 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:15:11.661727 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:11.661694 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:15:11.662216 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:11.661937 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" containerID="cri-o://306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc" gracePeriod=30 Sep 08 17:15:12.813019 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:12.812975 2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:15:12.813409 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:12.813053 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:15:16.920994 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:16.920968 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:15:17.098724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098630 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098766 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098794 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgtdd\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098827 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098877 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098915 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098949 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.098984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.098953 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:15:17.099491 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099013 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket\") pod \"1554d2d0-9775-499d-af71-815c1d3a721c\" (UID: \"1554d2d0-9775-499d-af71-815c1d3a721c\") " Sep 08 17:15:17.099491 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099423 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.099641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099611 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:15:17.099641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099627 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:15:17.099924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099848 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:15:17.099924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.099888 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:15:17.101168 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.101137 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:15:17.101265 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.101203 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:15:17.101437 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.101422 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:15:17.101493 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.101476 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd" (OuterVolumeSpecName: "kube-api-access-dgtdd") pod "1554d2d0-9775-499d-af71-815c1d3a721c" (UID: "1554d2d0-9775-499d-af71-815c1d3a721c"). InnerVolumeSpecName "kube-api-access-dgtdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:15:17.196636 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.196605 2556 generic.go:356] "Generic (PLEG): container finished" podID="1554d2d0-9775-499d-af71-815c1d3a721c" containerID="306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc" exitCode=0 Sep 08 17:15:17.196794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.196653 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" Sep 08 17:15:17.196794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.196692 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerDied","Data":"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc"} Sep 08 17:15:17.196888 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.196792 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5" event={"ID":"1554d2d0-9775-499d-af71-815c1d3a721c","Type":"ContainerDied","Data":"9a96a2ae736d3241761b870718f9cb697497ce7f8b1c6b6c0389b4a49c385f72"} Sep 08 17:15:17.196888 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.196822 2556 scope.go:117] "RemoveContainer" containerID="306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc" Sep 08 17:15:17.200644 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200585 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200644 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200643 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200653 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200664 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtdd\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-kube-api-access-dgtdd\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200678 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1554d2d0-9775-499d-af71-815c1d3a721c-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200691 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1554d2d0-9775-499d-af71-815c1d3a721c-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200703 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1554d2d0-9775-499d-af71-815c1d3a721c-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.200823 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.200717 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1554d2d0-9775-499d-af71-815c1d3a721c-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:15:17.206400 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.206377 2556 scope.go:117] "RemoveContainer" containerID="48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa" Sep 08 17:15:17.214152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.214137 2556 scope.go:117] "RemoveContainer" containerID="306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc" Sep 08 17:15:17.214466 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:15:17.214445 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc\": container with ID starting with 306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc not found: ID does not exist" containerID="306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc" Sep 08 17:15:17.214554 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.214473 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc"} err="failed to get container status \"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc\": rpc error: code = NotFound desc = could not find container \"306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc\": container with ID starting with 306662c1a55074acaf392e30a12e0276de2c447f02aaa76b8a343ccb0cc30dfc not found: ID does not exist" Sep 08 17:15:17.214554 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.214490 2556 scope.go:117] "RemoveContainer" containerID="48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa" Sep 08 17:15:17.214746 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:15:17.214730 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa\": container with ID starting with 48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa not found: ID does not exist" containerID="48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa" Sep 08 17:15:17.214807 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.214751 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa"} err="failed to get container status \"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa\": rpc error: code = NotFound desc = could not find container \"48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa\": container with ID starting with 48ac5c54adcd45c994bd4f48d6b4bae8b3da5d434895978cd2124932f24cc0aa not found: ID does not exist" Sep 08 17:15:17.221694 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.221674 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:15:17.225855 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.225834 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d7359b-openshift-default-7f954db468-4gdw5"] Sep 08 17:15:17.342413 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:17.342382 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" path="/var/lib/kubelet/pods/1554d2d0-9775-499d-af71-815c1d3a721c/volumes" Sep 08 17:15:34.106403 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:34.106364 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:15:49.410461 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:49.410423 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:15:54.926722 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.926685 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:15:54.927132 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927119 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927134 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927148 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5447325-2edf-4a2e-964e-5db88362534b" containerName="cleanup" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927154 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5447325-2edf-4a2e-964e-5db88362534b" containerName="cleanup" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927166 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927174 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927185 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f3ef02-3809-4f83-ae23-5930dc9aa7b7" containerName="cleanup" Sep 08 17:15:54.927195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927191 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f3ef02-3809-4f83-ae23-5930dc9aa7b7" containerName="cleanup" Sep 08 17:15:54.927475 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927253 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927475 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927263 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1554d2d0-9775-499d-af71-815c1d3a721c" containerName="istio-proxy" Sep 08 17:15:54.927475 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927272 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f3ef02-3809-4f83-ae23-5930dc9aa7b7" containerName="cleanup" Sep 08 17:15:54.927475 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.927280 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5447325-2edf-4a2e-964e-5db88362534b" containerName="cleanup" Sep 08 17:15:54.930670 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.930649 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:54.933560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.933536 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-978af9-openshift-default-dockercfg-bsfm6\"" Sep 08 17:15:54.942082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:54.942058 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:15:55.078602 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078564 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078602 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078604 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078637 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2x8q\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078711 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078731 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078801 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078968 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078831 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.078968 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.078885 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179393 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179437 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179471 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2x8q\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179527 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179548 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179586 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.179812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179685 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.180086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179872 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.180086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179910 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.180086 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.179955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.180235 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.180157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.180275 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.180260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.181891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.181862 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.182118 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.182100 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.188062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.188035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.188170 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.188157 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2x8q\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q\") pod \"e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.241179 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.241139 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:55.525875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.525843 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:15:55.527706 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:15:55.527681 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b8c2d7_e465_452b_996f_02427b3b3b30.slice/crio-cabb8166ede73eacae12d3e74e8bcdf36f8b51c94b755db277c08ac1235e600b WatchSource:0}: Error finding container cabb8166ede73eacae12d3e74e8bcdf36f8b51c94b755db277c08ac1235e600b: Status 404 returned error can't find the container with id cabb8166ede73eacae12d3e74e8bcdf36f8b51c94b755db277c08ac1235e600b Sep 08 17:15:55.529845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.529811 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:15:55.529954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.529891 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:15:55.529954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:55.529940 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:15:56.403940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:56.403904 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerStarted","Data":"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b"} Sep 08 17:15:56.404340 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:56.403980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerStarted","Data":"cabb8166ede73eacae12d3e74e8bcdf36f8b51c94b755db277c08ac1235e600b"} Sep 08 17:15:56.427565 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:56.427507 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podStartSLOduration=2.42748788 podStartE2EDuration="2.42748788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:15:56.426823897 +0000 UTC m=+1487.571594459" watchObservedRunningTime="2026-09-08 17:15:56.42748788 +0000 UTC m=+1487.572258441" Sep 08 17:15:57.241592 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:57.241556 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:57.246447 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:57.246421 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:57.411905 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:57.411869 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:15:57.413443 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:15:57.413422 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:16:00.469379 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.469345 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:16:00.472995 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.472975 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.475528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.475505 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-978af9-dockercfg-mhc2v\"" Sep 08 17:16:00.475628 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.475593 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-978af9\"" Sep 08 17:16:00.485875 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.485850 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:16:00.488378 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.488358 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:16:00.494916 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.494898 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.504405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.504379 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:16:00.533398 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.533370 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.533549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.533408 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.533549 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.533471 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vz6d\" (UniqueName: \"kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.533627 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.533555 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpqr\" (UniqueName: \"kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.634107 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpqr\" (UniqueName: \"kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.634280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634154 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.634280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.634399 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vz6d\" (UniqueName: \"kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.634902 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634881 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.635002 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.634985 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.642920 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.642895 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vz6d\" (UniqueName: \"kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d\") pod \"payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.644102 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.644079 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpqr\" (UniqueName: \"kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr\") pod \"payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:00.785454 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.785368 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:00.806642 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:00.806600 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:01.072458 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.072427 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:16:01.073965 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:16:01.073937 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0265f8c2_02eb_40b0_ba51_e15bee2767ae.slice/crio-1a26d97b2a2f4765081e511dd822e9666a0778cd702c27d17bc195067b7c70f4 WatchSource:0}: Error finding container 1a26d97b2a2f4765081e511dd822e9666a0778cd702c27d17bc195067b7c70f4: Status 404 returned error can't find the container with id 1a26d97b2a2f4765081e511dd822e9666a0778cd702c27d17bc195067b7c70f4 Sep 08 17:16:01.095917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.095892 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:16:01.098024 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:16:01.097976 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2864580e_0e8d_43d5_bc28_2ed81e8ed0b1.slice/crio-f3911b4e71f0f60ef8e2dd4258af9e217e894d1d6be63da7278f643c6b365979 WatchSource:0}: Error finding container f3911b4e71f0f60ef8e2dd4258af9e217e894d1d6be63da7278f643c6b365979: Status 404 returned error can't find the container with id f3911b4e71f0f60ef8e2dd4258af9e217e894d1d6be63da7278f643c6b365979 Sep 08 17:16:01.433044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.433003 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" event={"ID":"0265f8c2-02eb-40b0-ba51-e15bee2767ae","Type":"ContainerStarted","Data":"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2"} Sep 08 17:16:01.433248 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.433077 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:01.433248 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.433098 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" event={"ID":"0265f8c2-02eb-40b0-ba51-e15bee2767ae","Type":"ContainerStarted","Data":"1a26d97b2a2f4765081e511dd822e9666a0778cd702c27d17bc195067b7c70f4"} Sep 08 17:16:01.434546 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.434521 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" event={"ID":"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1","Type":"ContainerStarted","Data":"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e"} Sep 08 17:16:01.434641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.434563 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" event={"ID":"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1","Type":"ContainerStarted","Data":"f3911b4e71f0f60ef8e2dd4258af9e217e894d1d6be63da7278f643c6b365979"} Sep 08 17:16:01.434641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.434602 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:01.451076 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.451023 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" podStartSLOduration=1.451005382 podStartE2EDuration="1.451005382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:01.450483761 +0000 UTC m=+1492.595254325" watchObservedRunningTime="2026-09-08 17:16:01.451005382 +0000 UTC m=+1492.595775940" Sep 08 17:16:01.469657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:01.469602 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" podStartSLOduration=1.469582124 podStartE2EDuration="1.469582124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:01.467123227 +0000 UTC m=+1492.611893791" watchObservedRunningTime="2026-09-08 17:16:01.469582124 +0000 UTC m=+1492.614352686" Sep 08 17:16:02.902571 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:02.902526 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:16:09.624255 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:09.624221 2556 scope.go:117] "RemoveContainer" containerID="93c13aa702feb77e2e0076659caa08f226319aa5fa262219fa7e5bc488ee40dc" Sep 08 17:16:12.441998 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:12.441961 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:16:12.442385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:12.442124 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:16:19.553560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.553452 2556 generic.go:356] "Generic (PLEG): container finished" podID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerID="e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b" exitCode=0 Sep 08 17:16:19.553560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.553538 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerDied","Data":"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b"} Sep 08 17:16:19.554072 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.553984 2556 scope.go:117] "RemoveContainer" containerID="e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b" Sep 08 17:16:19.555073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.555044 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:19.555134 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.555111 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:19.555174 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:19.555141 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:20.567135 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:20.567099 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerStarted","Data":"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386"} Sep 08 17:16:21.241950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:21.241913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:16:21.242160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:21.242108 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 17:16:21.242237 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:21.242163 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 17:16:22.241790 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:22.241741 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 17:16:22.242270 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:22.241806 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 17:16:23.242374 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:23.242302 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 17:16:23.242877 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:23.242405 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 17:16:24.242218 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.242181 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 17:16:24.242480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.242239 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 17:16:24.325600 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.325569 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:16:24.328152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.328137 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.331719 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.331690 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a78a1e-openshift-default-dockercfg-h8fr9\"" Sep 08 17:16:24.362342 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.361056 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:16:24.392741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392698 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.392942 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.392942 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckgd\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.392942 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392883 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.392942 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392918 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.393102 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392963 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.393102 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.392998 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.393178 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.393098 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.393178 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.393145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494250 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494156 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494303 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckgd\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494420 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494449 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494519 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494581 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494876 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494761 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.494876 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494839 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.495020 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.494997 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.495160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.495141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.496779 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.496757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.496963 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.496946 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.504149 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.504127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckgd\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.504491 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.504469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token\") pod \"e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.639098 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.639070 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:24.936575 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.936547 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:16:24.938281 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:16:24.938252 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b2a09c_e2e7_4ec3_9545_ea3c11918c4f.slice/crio-43918c8ae990a02193bceccab4c1f7bc34b8d0369eb161356e571ee0d1b2770d WatchSource:0}: Error finding container 43918c8ae990a02193bceccab4c1f7bc34b8d0369eb161356e571ee0d1b2770d: Status 404 returned error can't find the container with id 43918c8ae990a02193bceccab4c1f7bc34b8d0369eb161356e571ee0d1b2770d Sep 08 17:16:24.940726 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.940697 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:24.940797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.940771 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:24.940840 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:24.940800 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:25.241649 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.241563 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 17:16:25.241649 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.241620 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 17:16:25.602254 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.602165 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerStarted","Data":"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181"} Sep 08 17:16:25.602254 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.602222 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerStarted","Data":"43918c8ae990a02193bceccab4c1f7bc34b8d0369eb161356e571ee0d1b2770d"} Sep 08 17:16:25.629056 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.628988 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podStartSLOduration=1.628964101 podStartE2EDuration="1.628964101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:25.625404658 +0000 UTC m=+1516.770175221" watchObservedRunningTime="2026-09-08 17:16:25.628964101 +0000 UTC m=+1516.773734666" Sep 08 17:16:25.639467 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:25.639434 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:26.246145 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:26.246117 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:16:26.246425 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:26.246404 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:16:26.247152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:26.247136 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:16:26.645593 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:26.645561 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:27.613538 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:27.613499 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:27.614527 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:27.614505 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:36.062588 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:36.062502 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:16:46.729006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.728971 2556 generic.go:356] "Generic (PLEG): container finished" podID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerID="7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181" exitCode=0 Sep 08 17:16:46.729447 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.729041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerDied","Data":"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181"} Sep 08 17:16:46.729528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.729514 2556 scope.go:117] "RemoveContainer" containerID="7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181" Sep 08 17:16:46.730592 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.730560 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:46.730705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.730643 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:46.730705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:46.730690 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:16:47.736264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:47.736226 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerStarted","Data":"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75"} Sep 08 17:16:48.639713 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:48.639672 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:48.639944 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:48.639921 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 08 17:16:48.640024 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:48.639980 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 08 17:16:49.640525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:49.640488 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 08 17:16:49.640935 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:49.640555 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 08 17:16:50.639637 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:50.639599 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 08 17:16:50.639816 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:50.639659 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 08 17:16:51.639698 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:51.639660 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 08 17:16:51.640159 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:51.639715 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 08 17:16:52.640240 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:52.640204 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Sep 08 17:16:52.640641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:52.640261 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Sep 08 17:16:53.643853 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:53.643822 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:53.644214 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:53.644180 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:16:53.644900 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:16:53.644883 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:17:09.648374 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:09.648340 2556 scope.go:117] "RemoveContainer" containerID="ec9a5a8cbb0dca95bc2784e24a9d80ebefd3155a0d86d5d88d8e0bddc47cf75e" Sep 08 17:17:15.006596 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:15.006554 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:17:28.320581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:28.320543 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:17:35.352058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:35.352027 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86_2864580e-0e8d-43d5-bc28-2ed81e8ed0b1/payload-processing/0.log" Sep 08 17:17:36.198366 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.198305 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:17:36.202247 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.202227 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.204985 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.204964 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-05b1b1-openshift-default-dockercfg-bc7vj\"" Sep 08 17:17:36.212761 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.212740 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:17:36.243761 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243725 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.243917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243779 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.243917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243860 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.243917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243891 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.243917 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243908 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.244065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.243969 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpts\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.244065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.244008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.244065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.244031 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.244065 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.244047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345278 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345243 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345278 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345282 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345301 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345388 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345434 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345537 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345560 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345612 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpts\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345637 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345985 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345834 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345985 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345880 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.345985 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.345958 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.346379 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.346356 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.348286 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.348266 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.348945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.348915 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.355994 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.355971 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.356249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.356192 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpts\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts\") pod \"e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.513919 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.513838 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:36.793234 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.793207 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:17:36.795364 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:17:36.795295 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1ed196_0ac8_4d0c_a874_ea6fc46ae1d2.slice/crio-04eea38a5aef777faf666ab6dcea79af5d16349c9def4d703587ae757758f911 WatchSource:0}: Error finding container 04eea38a5aef777faf666ab6dcea79af5d16349c9def4d703587ae757758f911: Status 404 returned error can't find the container with id 04eea38a5aef777faf666ab6dcea79af5d16349c9def4d703587ae757758f911 Sep 08 17:17:36.797849 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.797814 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:36.797921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.797883 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:36.797921 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:36.797912 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:37.022224 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:37.022186 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerStarted","Data":"4c54ab4150892b37843778f5230a97dcfb617acbec5fbe4b340f6b4114f35805"} Sep 08 17:17:37.022438 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:37.022247 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerStarted","Data":"04eea38a5aef777faf666ab6dcea79af5d16349c9def4d703587ae757758f911"} Sep 08 17:17:37.046495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:37.046389 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podStartSLOduration=1.046367327 podStartE2EDuration="1.046367327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:17:37.045593539 +0000 UTC m=+1588.190364100" watchObservedRunningTime="2026-09-08 17:17:37.046367327 +0000 UTC m=+1588.191137889" Sep 08 17:17:37.514379 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:37.514341 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:37.520473 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:37.520445 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:38.027905 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:38.027869 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:38.029011 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:38.028995 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:40.839430 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.839393 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:17:40.843098 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.843078 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:40.845824 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.845801 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-05b1b1\"" Sep 08 17:17:40.859657 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.859628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:17:40.895734 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.895699 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:40.895902 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.895740 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth7n\" (UniqueName: \"kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:40.998401 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.997946 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:40.998401 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.998012 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth7n\" (UniqueName: \"kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:41.000903 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:40.999408 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:41.011099 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:41.010968 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth7n\" (UniqueName: \"kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n\") pod \"payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:41.154021 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:41.153982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:41.433637 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:41.433611 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:17:41.434995 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:17:41.434969 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b38806_6c9a_4889_ae6f_bcbdbe0133ab.slice/crio-2331f258bd8fb026b89763e44055bad33ed0a191206faf42061fc56a28418e3e WatchSource:0}: Error finding container 2331f258bd8fb026b89763e44055bad33ed0a191206faf42061fc56a28418e3e: Status 404 returned error can't find the container with id 2331f258bd8fb026b89763e44055bad33ed0a191206faf42061fc56a28418e3e Sep 08 17:17:42.048457 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:42.048419 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" event={"ID":"36b38806-6c9a-4889-ae6f-bcbdbe0133ab","Type":"ContainerStarted","Data":"37a2b60d8bda2928a496967771ef2a2826ec986b6253d234e0806f4592e322a4"} Sep 08 17:17:42.048945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:42.048470 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" event={"ID":"36b38806-6c9a-4889-ae6f-bcbdbe0133ab","Type":"ContainerStarted","Data":"2331f258bd8fb026b89763e44055bad33ed0a191206faf42061fc56a28418e3e"} Sep 08 17:17:42.048945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:42.048547 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:42.069591 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:42.069533 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" podStartSLOduration=2.069513641 podStartE2EDuration="2.069513641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:17:42.066123085 +0000 UTC m=+1593.210893647" watchObservedRunningTime="2026-09-08 17:17:42.069513641 +0000 UTC m=+1593.214284204" Sep 08 17:17:47.226251 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:47.226215 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:17:53.057544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:53.057509 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:17:58.140037 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.140000 2556 generic.go:356] "Generic (PLEG): container finished" podID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerID="4c54ab4150892b37843778f5230a97dcfb617acbec5fbe4b340f6b4114f35805" exitCode=0 Sep 08 17:17:58.140544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.140059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerDied","Data":"4c54ab4150892b37843778f5230a97dcfb617acbec5fbe4b340f6b4114f35805"} Sep 08 17:17:58.140544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.140493 2556 scope.go:117] "RemoveContainer" containerID="4c54ab4150892b37843778f5230a97dcfb617acbec5fbe4b340f6b4114f35805" Sep 08 17:17:58.141711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.141677 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:58.141778 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.141748 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:58.141812 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:58.141781 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 08 17:17:59.152197 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:59.152153 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerStarted","Data":"46a1d2181d543f80eaa5de95ef98b90f45c4afbc9e51d9853ffaea46f69ffa35"} Sep 08 17:17:59.514695 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:59.514601 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:17:59.514892 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:59.514842 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 08 17:17:59.514972 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:17:59.514899 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 08 17:18:00.514711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:00.514671 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 08 17:18:00.515110 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:00.514728 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 08 17:18:01.514977 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:01.514938 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 08 17:18:01.515413 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:01.514993 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 08 17:18:02.515040 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:02.515006 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 08 17:18:02.515440 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:02.515063 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 08 17:18:03.514950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:03.514914 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 08 17:18:03.515143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:03.514972 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 08 17:18:04.519375 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:04.519345 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:18:04.519811 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:04.519789 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:18:04.520486 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:04.520467 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:18:10.981605 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:10.981571 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:18:10.982091 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:10.981855 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" podUID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" containerName="payload-processing" containerID="cri-o://37a2b60d8bda2928a496967771ef2a2826ec986b6253d234e0806f4592e322a4" gracePeriod=30 Sep 08 17:18:11.227134 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.227103 2556 generic.go:356] "Generic (PLEG): container finished" podID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" containerID="37a2b60d8bda2928a496967771ef2a2826ec986b6253d234e0806f4592e322a4" exitCode=0 Sep 08 17:18:11.227283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.227180 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" event={"ID":"36b38806-6c9a-4889-ae6f-bcbdbe0133ab","Type":"ContainerDied","Data":"37a2b60d8bda2928a496967771ef2a2826ec986b6253d234e0806f4592e322a4"} Sep 08 17:18:11.243821 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.243758 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:18:11.324242 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.324201 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lth7n\" (UniqueName: \"kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n\") pod \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " Sep 08 17:18:11.324455 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.324382 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume\") pod \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\" (UID: \"36b38806-6c9a-4889-ae6f-bcbdbe0133ab\") " Sep 08 17:18:11.325166 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.325137 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36b38806-6c9a-4889-ae6f-bcbdbe0133ab" (UID: "36b38806-6c9a-4889-ae6f-bcbdbe0133ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:11.341767 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.341718 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n" (OuterVolumeSpecName: "kube-api-access-lth7n") pod "36b38806-6c9a-4889-ae6f-bcbdbe0133ab" (UID: "36b38806-6c9a-4889-ae6f-bcbdbe0133ab"). InnerVolumeSpecName "kube-api-access-lth7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:11.426362 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.426311 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:11.426362 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:11.426355 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lth7n\" (UniqueName: \"kubernetes.io/projected/36b38806-6c9a-4889-ae6f-bcbdbe0133ab-kube-api-access-lth7n\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:12.233925 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:12.233894 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" Sep 08 17:18:12.233925 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:12.233906 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc" event={"ID":"36b38806-6c9a-4889-ae6f-bcbdbe0133ab","Type":"ContainerDied","Data":"2331f258bd8fb026b89763e44055bad33ed0a191206faf42061fc56a28418e3e"} Sep 08 17:18:12.234498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:12.233993 2556 scope.go:117] "RemoveContainer" containerID="37a2b60d8bda2928a496967771ef2a2826ec986b6253d234e0806f4592e322a4" Sep 08 17:18:12.252514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:12.252488 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:18:12.258284 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:12.258261 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-05b1b1-67644c7f4f-c2dqc"] Sep 08 17:18:13.344073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:13.344031 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" path="/var/lib/kubelet/pods/36b38806-6c9a-4889-ae6f-bcbdbe0133ab/volumes" Sep 08 17:18:15.312900 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:15.312864 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:18:18.174302 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:18.174266 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:18:18.174690 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:18.174539 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" containerID="cri-o://46a1d2181d543f80eaa5de95ef98b90f45c4afbc9e51d9853ffaea46f69ffa35" gracePeriod=30 Sep 08 17:18:19.521655 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:19.521622 2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:18:19.522025 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:19.521679 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:18:23.296034 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.296003 2556 generic.go:356] "Generic (PLEG): container finished" podID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerID="46a1d2181d543f80eaa5de95ef98b90f45c4afbc9e51d9853ffaea46f69ffa35" exitCode=0 Sep 08 17:18:23.296431 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.296057 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerDied","Data":"46a1d2181d543f80eaa5de95ef98b90f45c4afbc9e51d9853ffaea46f69ffa35"} Sep 08 17:18:23.296431 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.296104 2556 scope.go:117] "RemoveContainer" containerID="4c54ab4150892b37843778f5230a97dcfb617acbec5fbe4b340f6b4114f35805" Sep 08 17:18:23.434954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.434896 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:18:23.560817 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560779 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560849 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560872 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560916 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560944 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561001 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.560985 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561046 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561050 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:18:23.561271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561100 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khpts\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561156 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token\") pod \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\" (UID: \"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2\") " Sep 08 17:18:23.561271 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561218 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:18:23.561578 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561351 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:18:23.561578 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561380 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:18:23.561578 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561400 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:23.561741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561601 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.561741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561617 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.561741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561626 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.561741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561635 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.561741 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.561643 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.563081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.563058 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:18:23.563290 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.563271 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:23.563579 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.563559 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:18:23.563633 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.563583 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts" (OuterVolumeSpecName: "kube-api-access-khpts") pod "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" (UID: "4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2"). InnerVolumeSpecName "kube-api-access-khpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:23.662736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.662701 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.662736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.662730 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khpts\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-kube-api-access-khpts\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.662736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.662741 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:23.662969 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:23.662749 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:24.303270 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:24.303245 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" Sep 08 17:18:24.303732 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:24.303238 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq" event={"ID":"4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2","Type":"ContainerDied","Data":"04eea38a5aef777faf666ab6dcea79af5d16349c9def4d703587ae757758f911"} Sep 08 17:18:24.303732 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:24.303383 2556 scope.go:117] "RemoveContainer" containerID="46a1d2181d543f80eaa5de95ef98b90f45c4afbc9e51d9853ffaea46f69ffa35" Sep 08 17:18:24.327939 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:24.327910 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:18:24.329964 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:24.329942 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-05b1b1-openshift-default-79df668986-pz5jq"] Sep 08 17:18:25.342908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:25.342875 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" path="/var/lib/kubelet/pods/4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2/volumes" Sep 08 17:18:34.919137 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:34.919090 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:18:37.413010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.412975 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:18:37.413476 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.413233 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" podUID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" containerName="payload-processing" containerID="cri-o://adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e" gracePeriod=30 Sep 08 17:18:37.430469 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.430416 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:18:37.430719 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.430694 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" podUID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" containerName="payload-pre-processing" containerID="cri-o://cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2" gracePeriod=30 Sep 08 17:18:37.687635 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.687609 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:18:37.717762 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.717735 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:18:37.801742 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.801706 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume\") pod \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " Sep 08 17:18:37.801927 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.801778 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vz6d\" (UniqueName: \"kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d\") pod \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\" (UID: \"0265f8c2-02eb-40b0-ba51-e15bee2767ae\") " Sep 08 17:18:37.801985 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.801948 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjpqr\" (UniqueName: \"kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr\") pod \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " Sep 08 17:18:37.802047 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.802032 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume\") pod \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\" (UID: \"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1\") " Sep 08 17:18:37.802134 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.802109 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0265f8c2-02eb-40b0-ba51-e15bee2767ae" (UID: "0265f8c2-02eb-40b0-ba51-e15bee2767ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:37.802437 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.802409 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" (UID: "2864580e-0e8d-43d5-bc28-2ed81e8ed0b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:18:37.802560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.802506 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0265f8c2-02eb-40b0-ba51-e15bee2767ae-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.802560 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.802525 2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-plugins-config-volume\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.804527 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.804508 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr" (OuterVolumeSpecName: "kube-api-access-tjpqr") pod "2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" (UID: "2864580e-0e8d-43d5-bc28-2ed81e8ed0b1"). InnerVolumeSpecName "kube-api-access-tjpqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:37.804527 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.804515 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d" (OuterVolumeSpecName: "kube-api-access-2vz6d") pod "0265f8c2-02eb-40b0-ba51-e15bee2767ae" (UID: "0265f8c2-02eb-40b0-ba51-e15bee2767ae"). InnerVolumeSpecName "kube-api-access-2vz6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:18:37.903316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.903274 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vz6d\" (UniqueName: \"kubernetes.io/projected/0265f8c2-02eb-40b0-ba51-e15bee2767ae-kube-api-access-2vz6d\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:37.903316 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:37.903308 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpqr\" (UniqueName: \"kubernetes.io/projected/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1-kube-api-access-tjpqr\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:18:38.387587 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.387552 2556 generic.go:356] "Generic (PLEG): container finished" podID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" containerID="adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e" exitCode=0 Sep 08 17:18:38.387794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.387606 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" Sep 08 17:18:38.387794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.387632 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" event={"ID":"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1","Type":"ContainerDied","Data":"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e"} Sep 08 17:18:38.387794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.387682 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86" event={"ID":"2864580e-0e8d-43d5-bc28-2ed81e8ed0b1","Type":"ContainerDied","Data":"f3911b4e71f0f60ef8e2dd4258af9e217e894d1d6be63da7278f643c6b365979"} Sep 08 17:18:38.387794 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.387709 2556 scope.go:117] "RemoveContainer" containerID="adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e" Sep 08 17:18:38.389123 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.389100 2556 generic.go:356] "Generic (PLEG): container finished" podID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" containerID="cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2" exitCode=0 Sep 08 17:18:38.389223 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.389135 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" Sep 08 17:18:38.389223 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.389194 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" event={"ID":"0265f8c2-02eb-40b0-ba51-e15bee2767ae","Type":"ContainerDied","Data":"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2"} Sep 08 17:18:38.389368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.389236 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb" event={"ID":"0265f8c2-02eb-40b0-ba51-e15bee2767ae","Type":"ContainerDied","Data":"1a26d97b2a2f4765081e511dd822e9666a0778cd702c27d17bc195067b7c70f4"} Sep 08 17:18:38.402087 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.402070 2556 scope.go:117] "RemoveContainer" containerID="adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e" Sep 08 17:18:38.402399 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:18:38.402381 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e\": container with ID starting with adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e not found: ID does not exist" containerID="adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e" Sep 08 17:18:38.402498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.402406 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e"} err="failed to get container status \"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e\": rpc error: code = NotFound desc = could not find container \"adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e\": container with ID starting with adec327bb60d11c5a72cbd98e15db21ff63183836a952b5c95027147e8d81a0e not found: ID does not exist" Sep 08 17:18:38.402498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.402423 2556 scope.go:117] "RemoveContainer" containerID="cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2" Sep 08 17:18:38.410680 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.410661 2556 scope.go:117] "RemoveContainer" containerID="cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2" Sep 08 17:18:38.410995 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:18:38.410977 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2\": container with ID starting with cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2 not found: ID does not exist" containerID="cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2" Sep 08 17:18:38.411056 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.411001 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2"} err="failed to get container status \"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2\": rpc error: code = NotFound desc = could not find container \"cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2\": container with ID starting with cdd392d0d63f42e1b36160bd8ced98e6879788e3c784c3dd4681ce1be9d625e2 not found: ID does not exist" Sep 08 17:18:38.416554 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.416527 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:18:38.420219 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.420198 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-978af9-6c5566686-4t5fb"] Sep 08 17:18:38.435424 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.435394 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:18:38.437252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:38.437227 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-978af9-6c4b949857-9gl86"] Sep 08 17:18:39.342799 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:39.342765 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" path="/var/lib/kubelet/pods/0265f8c2-02eb-40b0-ba51-e15bee2767ae/volumes" Sep 08 17:18:39.343141 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:39.343128 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" path="/var/lib/kubelet/pods/2864580e-0e8d-43d5-bc28-2ed81e8ed0b1/volumes" Sep 08 17:18:47.721101 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:18:47.721063 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:19:26.766910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:26.766878 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:19:26.767573 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:26.767112 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" containerID="cri-o://38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386" gracePeriod=30 Sep 08 17:19:27.129419 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129378 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst"] Sep 08 17:19:27.129891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129871 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.129891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129891 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129903 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" containerName="payload-processing" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129911 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" containerName="payload-processing" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129926 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" containerName="payload-processing" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129934 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" containerName="payload-processing" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129942 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129950 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129986 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" containerName="payload-pre-processing" Sep 08 17:19:27.130052 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.129993 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" containerName="payload-pre-processing" Sep 08 17:19:27.130485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.130097 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.130485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.130110 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2864580e-0e8d-43d5-bc28-2ed81e8ed0b1" containerName="payload-processing" Sep 08 17:19:27.130485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.130121 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1ed196-0ac8-4d0c-a874-ea6fc46ae1d2" containerName="istio-proxy" Sep 08 17:19:27.130485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.130131 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0265f8c2-02eb-40b0-ba51-e15bee2767ae" containerName="payload-pre-processing" Sep 08 17:19:27.130485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.130142 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b38806-6c9a-4889-ae6f-bcbdbe0133ab" containerName="payload-processing" Sep 08 17:19:27.133263 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.133243 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:27.136123 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.136090 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:19:27.136396 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.136382 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:19:27.144353 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.144298 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst"] Sep 08 17:19:27.229048 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.229013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst\" (UID: \"a70e3313-9df5-4cfd-8d5d-9099fa2507f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:27.329795 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.329760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst\" (UID: \"a70e3313-9df5-4cfd-8d5d-9099fa2507f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:27.330388 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.330368 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst\" (UID: \"a70e3313-9df5-4cfd-8d5d-9099fa2507f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:27.444986 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.444906 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:27.721479 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:27.721437 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst"] Sep 08 17:19:27.724200 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:19:27.724171 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70e3313_9df5_4cfd_8d5d_9099fa2507f4.slice/crio-f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973 WatchSource:0}: Error finding container f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973: Status 404 returned error can't find the container with id f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973 Sep 08 17:19:28.657683 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:28.657651 2556 generic.go:356] "Generic (PLEG): container finished" podID="a70e3313-9df5-4cfd-8d5d-9099fa2507f4" containerID="69120462bd44ecfad988027ff9d23ce66c27595596cf11613196a79def50cfdf" exitCode=0 Sep 08 17:19:28.658075 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:28.657747 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" event={"ID":"a70e3313-9df5-4cfd-8d5d-9099fa2507f4","Type":"ContainerDied","Data":"69120462bd44ecfad988027ff9d23ce66c27595596cf11613196a79def50cfdf"} Sep 08 17:19:28.658075 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:28.657796 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" event={"ID":"a70e3313-9df5-4cfd-8d5d-9099fa2507f4","Type":"ContainerStarted","Data":"f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973"} Sep 08 17:19:29.796526 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:29.796497 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:29.852938 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:29.852903 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca\") pod \"a70e3313-9df5-4cfd-8d5d-9099fa2507f4\" (UID: \"a70e3313-9df5-4cfd-8d5d-9099fa2507f4\") " Sep 08 17:19:29.853246 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:29.853218 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a70e3313-9df5-4cfd-8d5d-9099fa2507f4" (UID: "a70e3313-9df5-4cfd-8d5d-9099fa2507f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:29.853368 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:29.853354 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e3313-9df5-4cfd-8d5d-9099fa2507f4-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:30.669714 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:30.669681 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" event={"ID":"a70e3313-9df5-4cfd-8d5d-9099fa2507f4","Type":"ContainerDied","Data":"f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973"} Sep 08 17:19:30.669714 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:30.669703 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst" Sep 08 17:19:30.669931 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:30.669730 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55dcdaae59fbc0ce3fe2af36402896dbdc7d9db7631cabccedd6d23f38b3973" Sep 08 17:19:32.006932 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.006904 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:19:32.076394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076276 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076316 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2x8q\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076394 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076376 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076400 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076433 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076472 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076560 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076589 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076598 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:32.076674 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076634 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert\") pod \"e1b8c2d7-e465-452b-996f-02427b3b3b30\" (UID: \"e1b8c2d7-e465-452b-996f-02427b3b3b30\") " Sep 08 17:19:32.077005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076711 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:32.077005 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076725 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:32.077109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.077009 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.077109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.077029 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.077109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.077044 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.077109 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.076998 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:32.077272 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.077153 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:32.078703 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.078675 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q" (OuterVolumeSpecName: "kube-api-access-z2x8q") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "kube-api-access-z2x8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:32.078821 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.078740 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:19:32.079031 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.079007 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:32.079122 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.079035 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1b8c2d7-e465-452b-996f-02427b3b3b30" (UID: "e1b8c2d7-e465-452b-996f-02427b3b3b30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:32.177810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177772 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1b8c2d7-e465-452b-996f-02427b3b3b30-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.177810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177802 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2x8q\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-kube-api-access-z2x8q\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.177810 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177813 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.178044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177822 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.178044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177831 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1b8c2d7-e465-452b-996f-02427b3b3b30-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.178044 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.177838 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1b8c2d7-e465-452b-996f-02427b3b3b30-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:32.681377 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.681340 2556 generic.go:356] "Generic (PLEG): container finished" podID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerID="38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386" exitCode=0 Sep 08 17:19:32.681581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.681402 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" Sep 08 17:19:32.681581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.681422 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerDied","Data":"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386"} Sep 08 17:19:32.681581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.681482 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr" event={"ID":"e1b8c2d7-e465-452b-996f-02427b3b3b30","Type":"ContainerDied","Data":"cabb8166ede73eacae12d3e74e8bcdf36f8b51c94b755db277c08ac1235e600b"} Sep 08 17:19:32.681581 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.681510 2556 scope.go:117] "RemoveContainer" containerID="38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386" Sep 08 17:19:32.690913 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.690896 2556 scope.go:117] "RemoveContainer" containerID="e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b" Sep 08 17:19:32.698754 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.698737 2556 scope.go:117] "RemoveContainer" containerID="38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386" Sep 08 17:19:32.699014 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:19:32.698996 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386\": container with ID starting with 38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386 not found: ID does not exist" containerID="38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386" Sep 08 17:19:32.699058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.699022 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386"} err="failed to get container status \"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386\": rpc error: code = NotFound desc = could not find container \"38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386\": container with ID starting with 38ec26f60dceb387a99cd56a2d0bf1638ed8c76f68f7b60579bdde0e5c2d7386 not found: ID does not exist" Sep 08 17:19:32.699058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.699040 2556 scope.go:117] "RemoveContainer" containerID="e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b" Sep 08 17:19:32.699278 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:19:32.699259 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b\": container with ID starting with e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b not found: ID does not exist" containerID="e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b" Sep 08 17:19:32.699350 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.699284 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b"} err="failed to get container status \"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b\": rpc error: code = NotFound desc = could not find container \"e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b\": container with ID starting with e814c07e52612b6fa7fdde352b012d3a3f82a43fd681aecbe8e67d4ebb243f4b not found: ID does not exist" Sep 08 17:19:32.706688 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.706666 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:19:32.711280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:32.711256 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-978af9-openshift-default-86746767f8-gp2wr"] Sep 08 17:19:33.343820 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:33.343787 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" path="/var/lib/kubelet/pods/e1b8c2d7-e465-452b-996f-02427b3b3b30/volumes" Sep 08 17:19:38.008920 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:38.008886 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst"] Sep 08 17:19:38.015136 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:38.015106 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a78a1e-4tsst"] Sep 08 17:19:39.343470 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:39.343440 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70e3313-9df5-4cfd-8d5d-9099fa2507f4" path="/var/lib/kubelet/pods/a70e3313-9df5-4cfd-8d5d-9099fa2507f4/volumes" Sep 08 17:19:39.908238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:39.908172 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:19:44.793487 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:44.793451 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:19:44.794096 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:44.793658 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" containerID="cri-o://1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75" gracePeriod=30 Sep 08 17:19:45.173310 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173278 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr"] Sep 08 17:19:45.173735 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173719 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:19:45.173781 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173737 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:19:45.173781 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173745 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70e3313-9df5-4cfd-8d5d-9099fa2507f4" containerName="revoke-keys" Sep 08 17:19:45.173781 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173750 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70e3313-9df5-4cfd-8d5d-9099fa2507f4" containerName="revoke-keys" Sep 08 17:19:45.173879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173834 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:19:45.173879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.173844 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70e3313-9df5-4cfd-8d5d-9099fa2507f4" containerName="revoke-keys" Sep 08 17:19:45.177192 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.177176 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:45.179782 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.179760 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:19:45.180028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.180009 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:19:45.186238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.186216 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr"] Sep 08 17:19:45.307757 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.307718 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr\" (UID: \"fe8d47c3-cf9e-4149-8183-d532343d51f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:45.409152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.409115 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr\" (UID: \"fe8d47c3-cf9e-4149-8183-d532343d51f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:45.409796 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.409777 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr\" (UID: \"fe8d47c3-cf9e-4149-8183-d532343d51f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:45.489385 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.489285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:45.763935 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:45.763906 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr"] Sep 08 17:19:45.766398 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:19:45.766373 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8d47c3_cf9e_4149_8183_d532343d51f2.slice/crio-7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5 WatchSource:0}: Error finding container 7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5: Status 404 returned error can't find the container with id 7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5 Sep 08 17:19:46.757574 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:46.757479 2556 generic.go:356] "Generic (PLEG): container finished" podID="fe8d47c3-cf9e-4149-8183-d532343d51f2" containerID="dbd8b1f648ce18eb43829e18a57a491bbcf28b7c51f62f3f4caa4c16be1d2e17" exitCode=0 Sep 08 17:19:46.757574 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:46.757556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" event={"ID":"fe8d47c3-cf9e-4149-8183-d532343d51f2","Type":"ContainerDied","Data":"dbd8b1f648ce18eb43829e18a57a491bbcf28b7c51f62f3f4caa4c16be1d2e17"} Sep 08 17:19:46.758058 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:46.757603 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" event={"ID":"fe8d47c3-cf9e-4149-8183-d532343d51f2","Type":"ContainerStarted","Data":"7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5"} Sep 08 17:19:47.894585 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:47.894561 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:48.037442 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.037350 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca\") pod \"fe8d47c3-cf9e-4149-8183-d532343d51f2\" (UID: \"fe8d47c3-cf9e-4149-8183-d532343d51f2\") " Sep 08 17:19:48.037752 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.037726 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe8d47c3-cf9e-4149-8183-d532343d51f2" (UID: "fe8d47c3-cf9e-4149-8183-d532343d51f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:48.138667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.138630 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8d47c3-cf9e-4149-8183-d532343d51f2-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:48.770145 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.770115 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" Sep 08 17:19:48.770341 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.770122 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr" event={"ID":"fe8d47c3-cf9e-4149-8183-d532343d51f2","Type":"ContainerDied","Data":"7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5"} Sep 08 17:19:48.770341 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:48.770227 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7085704f8f299b09481161d693a88855f72335a7150db27eee02fa4ae04d93c5" Sep 08 17:19:50.037336 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.037300 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:19:50.159804 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159755 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159824 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159867 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159882 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159902 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160010 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.159917 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160038 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160059 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160078 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rckgd\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160130 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy\") pod \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\" (UID: \"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f\") " Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160192 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:50.160274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160265 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:19:50.160645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160281 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data" (OuterVolumeSpecName: "istio-data") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:50.160645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160515 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.160645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160534 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.160645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160548 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.160645 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160560 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.160910 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.160867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:50.162252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.162206 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:50.162252 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.162225 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:19:50.162570 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.162546 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd" (OuterVolumeSpecName: "kube-api-access-rckgd") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "kube-api-access-rckgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:19:50.162638 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.162553 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" (UID: "80b2a09c-e2e7-4ec3-9545-ea3c11918c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:19:50.262062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.262028 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.262062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.262057 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.262062 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.262066 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rckgd\" (UniqueName: \"kubernetes.io/projected/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-kube-api-access-rckgd\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.262295 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.262075 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.262295 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.262083 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:19:50.782184 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.782144 2556 generic.go:356] "Generic (PLEG): container finished" podID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerID="1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75" exitCode=0 Sep 08 17:19:50.782420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.782200 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" Sep 08 17:19:50.782420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.782233 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerDied","Data":"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75"} Sep 08 17:19:50.782420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.782302 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw" event={"ID":"80b2a09c-e2e7-4ec3-9545-ea3c11918c4f","Type":"ContainerDied","Data":"43918c8ae990a02193bceccab4c1f7bc34b8d0369eb161356e571ee0d1b2770d"} Sep 08 17:19:50.782420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.782357 2556 scope.go:117] "RemoveContainer" containerID="1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75" Sep 08 17:19:50.791693 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.791675 2556 scope.go:117] "RemoveContainer" containerID="7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181" Sep 08 17:19:50.799470 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.799450 2556 scope.go:117] "RemoveContainer" containerID="1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75" Sep 08 17:19:50.799721 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:19:50.799697 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75\": container with ID starting with 1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75 not found: ID does not exist" containerID="1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75" Sep 08 17:19:50.799792 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.799732 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75"} err="failed to get container status \"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75\": rpc error: code = NotFound desc = could not find container \"1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75\": container with ID starting with 1cfd1b0b4f1df642726b7dc85b8d8a8c39dee8f0a67fef1797bc097386e8fc75 not found: ID does not exist" Sep 08 17:19:50.799792 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.799756 2556 scope.go:117] "RemoveContainer" containerID="7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181" Sep 08 17:19:50.800003 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:19:50.799988 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181\": container with ID starting with 7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181 not found: ID does not exist" containerID="7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181" Sep 08 17:19:50.800045 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.800008 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181"} err="failed to get container status \"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181\": rpc error: code = NotFound desc = could not find container \"7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181\": container with ID starting with 7ee6fb2138ca31c584019eba0209120dc87e1ac37c4c72ec0a206174f6ca6181 not found: ID does not exist" Sep 08 17:19:50.807728 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.807516 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:19:50.809732 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:50.809707 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a78a1e-openshift-default-6ddfcb846b-wl2bw"] Sep 08 17:19:51.343765 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:51.343729 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" path="/var/lib/kubelet/pods/80b2a09c-e2e7-4ec3-9545-ea3c11918c4f/volumes" Sep 08 17:19:55.991929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:55.991896 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr"] Sep 08 17:19:55.997879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:55.997852 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c05c9f-pswmr"] Sep 08 17:19:57.342308 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:57.342274 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8d47c3-cf9e-4149-8183-d532343d51f2" path="/var/lib/kubelet/pods/fe8d47c3-cf9e-4149-8183-d532343d51f2/volumes" Sep 08 17:19:58.730121 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:19:58.730084 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:20:02.775701 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:02.775666 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:20:02.776176 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:02.775896 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" containerID="cri-o://f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f" gracePeriod=30 Sep 08 17:20:03.146731 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.146691 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z"] Sep 08 17:20:03.147186 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147169 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8d47c3-cf9e-4149-8183-d532343d51f2" containerName="revoke-keys" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147190 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8d47c3-cf9e-4149-8183-d532343d51f2" containerName="revoke-keys" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147204 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147213 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147251 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147260 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147275 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.147297 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147283 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.147667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147425 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b8c2d7-e465-452b-996f-02427b3b3b30" containerName="istio-proxy" Sep 08 17:20:03.147667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147439 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.147667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147450 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8d47c3-cf9e-4149-8183-d532343d51f2" containerName="revoke-keys" Sep 08 17:20:03.147667 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.147459 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="80b2a09c-e2e7-4ec3-9545-ea3c11918c4f" containerName="istio-proxy" Sep 08 17:20:03.152195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.152175 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:03.154848 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.154827 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\"" Sep 08 17:20:03.155099 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.155079 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:20:03.163841 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.163817 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z"] Sep 08 17:20:03.179403 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.179377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z\" (UID: \"70633eae-2719-4899-a1f8-d232c0784dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:03.280845 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.280808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z\" (UID: \"70633eae-2719-4899-a1f8-d232c0784dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:03.281454 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.281436 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z\" (UID: \"70633eae-2719-4899-a1f8-d232c0784dce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:03.463238 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.463141 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:03.737747 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.737721 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z"] Sep 08 17:20:03.740276 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:20:03.740248 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70633eae_2719_4899_a1f8_d232c0784dce.slice/crio-44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3 WatchSource:0}: Error finding container 44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3: Status 404 returned error can't find the container with id 44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3 Sep 08 17:20:03.852941 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:03.852910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" event={"ID":"70633eae-2719-4899-a1f8-d232c0784dce","Type":"ContainerStarted","Data":"44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3"} Sep 08 17:20:04.859931 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:04.859838 2556 generic.go:356] "Generic (PLEG): container finished" podID="70633eae-2719-4899-a1f8-d232c0784dce" containerID="88805043720b2ee8a5a4fe873fc0e137566087468e6f3642947449d3ec744bcc" exitCode=0 Sep 08 17:20:04.859931 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:04.859878 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" event={"ID":"70633eae-2719-4899-a1f8-d232c0784dce","Type":"ContainerDied","Data":"88805043720b2ee8a5a4fe873fc0e137566087468e6f3642947449d3ec744bcc"} Sep 08 17:20:05.988767 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:05.988733 2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 17:20:05.989084 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:05.988802 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 17:20:05.999658 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:05.999635 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:06.002954 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.002930 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca\") pod \"70633eae-2719-4899-a1f8-d232c0784dce\" (UID: \"70633eae-2719-4899-a1f8-d232c0784dce\") " Sep 08 17:20:06.003232 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.003210 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70633eae-2719-4899-a1f8-d232c0784dce" (UID: "70633eae-2719-4899-a1f8-d232c0784dce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:20:06.103788 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.103757 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70633eae-2719-4899-a1f8-d232c0784dce-maas-api-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:06.873607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.873579 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" Sep 08 17:20:06.873607 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.873594 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z" event={"ID":"70633eae-2719-4899-a1f8-d232c0784dce","Type":"ContainerDied","Data":"44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3"} Sep 08 17:20:06.873874 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:06.873638 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c7a17d262e93f566fd343c9cfbe7f85cb6468f939f8b233de2bc7e80f328a3" Sep 08 17:20:08.037408 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.037383 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:20:08.122081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122047 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122107 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122160 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122222 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122274 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122250 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122275 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122310 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbvqw\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122376 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122400 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122430 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo\") pod \"1cd354ff-1818-430c-9abd-0d4135f952ff\" (UID: \"1cd354ff-1818-430c-9abd-0d4135f952ff\") " Sep 08 17:20:08.122525 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122480 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:08.122826 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122600 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:08.122826 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122720 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:08.122826 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122751 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:20:08.122967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122840 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd354ff-1818-430c-9abd-0d4135f952ff-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.122967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122856 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.122967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122866 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.122967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122874 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.122967 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.122882 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.124572 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.124548 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:20:08.124572 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.124561 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw" (OuterVolumeSpecName: "kube-api-access-fbvqw") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "kube-api-access-fbvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:08.124818 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.124797 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:08.124879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.124799 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cd354ff-1818-430c-9abd-0d4135f952ff" (UID: "1cd354ff-1818-430c-9abd-0d4135f952ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:08.223934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.223899 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.223934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.223928 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.223934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.223937 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvqw\" (UniqueName: \"kubernetes.io/projected/1cd354ff-1818-430c-9abd-0d4135f952ff-kube-api-access-fbvqw\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.223934 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.223946 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd354ff-1818-430c-9abd-0d4135f952ff-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:08.886678 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.886643 2556 generic.go:356] "Generic (PLEG): container finished" podID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerID="f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f" exitCode=0 Sep 08 17:20:08.886895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.886698 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" Sep 08 17:20:08.886895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.886738 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerDied","Data":"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f"} Sep 08 17:20:08.886895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.886803 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn" event={"ID":"1cd354ff-1818-430c-9abd-0d4135f952ff","Type":"ContainerDied","Data":"bc9f9f86b9e8b88b22c473d42705e533d2d09e87478488612f066c8dc195a047"} Sep 08 17:20:08.886895 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.886833 2556 scope.go:117] "RemoveContainer" containerID="f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f" Sep 08 17:20:08.896734 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.896581 2556 scope.go:117] "RemoveContainer" containerID="4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993" Sep 08 17:20:08.906814 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.906793 2556 scope.go:117] "RemoveContainer" containerID="f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f" Sep 08 17:20:08.907885 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:20:08.907859 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f\": container with ID starting with f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f not found: ID does not exist" containerID="f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f" Sep 08 17:20:08.907971 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.907891 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f"} err="failed to get container status \"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f\": rpc error: code = NotFound desc = could not find container \"f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f\": container with ID starting with f256dbf61a14072d1c13b4870d38b0f83185b4ac7981013d018056e55a3bfc4f not found: ID does not exist" Sep 08 17:20:08.907971 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.907908 2556 scope.go:117] "RemoveContainer" containerID="4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993" Sep 08 17:20:08.908214 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:20:08.908193 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993\": container with ID starting with 4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993 not found: ID does not exist" containerID="4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993" Sep 08 17:20:08.908259 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.908218 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993"} err="failed to get container status \"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993\": rpc error: code = NotFound desc = could not find container \"4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993\": container with ID starting with 4bf25f5d00052aa5e812e5ea99e73c3fc0cfefa235ce89022d4a772778087993 not found: ID does not exist" Sep 08 17:20:08.909504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.909485 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:20:08.913626 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:08.913605 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c05c9f-openshift-default-bc85fb448-484kn"] Sep 08 17:20:09.342720 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:09.342638 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" path="/var/lib/kubelet/pods/1cd354ff-1818-430c-9abd-0d4135f952ff/volumes" Sep 08 17:20:09.699283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:09.699251 2556 scope.go:117] "RemoveContainer" containerID="588ca5b78946548f1992edb8970fd7c199f04f84b367ba48c88cd121b6809b1d" Sep 08 17:20:13.674719 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:13.674687 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26"] Sep 08 17:20:13.677305 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:13.677275 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-338600-29814795-xbc26"] Sep 08 17:20:13.900378 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:13.900341 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z"] Sep 08 17:20:13.905400 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:13.905366 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-338600-vz74z"] Sep 08 17:20:15.348589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:15.348555 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70633eae-2719-4899-a1f8-d232c0784dce" path="/var/lib/kubelet/pods/70633eae-2719-4899-a1f8-d232c0784dce/volumes" Sep 08 17:20:15.349013 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:15.348848 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f3ef02-3809-4f83-ae23-5930dc9aa7b7" path="/var/lib/kubelet/pods/c0f3ef02-3809-4f83-ae23-5930dc9aa7b7/volumes" Sep 08 17:20:16.213925 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:16.213885 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:20:20.790891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:20.790856 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:20:20.791347 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:20.791066 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" containerID="cri-o://a280a45534628d100cc51f96ba9cf664a8576f91313cd4eea58f6b63fa4e1f0d" gracePeriod=30 Sep 08 17:20:25.410143 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:25.410105 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:20:25.972452 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:25.972422 2556 generic.go:356] "Generic (PLEG): container finished" podID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerID="a280a45534628d100cc51f96ba9cf664a8576f91313cd4eea58f6b63fa4e1f0d" exitCode=0 Sep 08 17:20:25.972590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:25.972500 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerDied","Data":"a280a45534628d100cc51f96ba9cf664a8576f91313cd4eea58f6b63fa4e1f0d"} Sep 08 17:20:25.972590 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:25.972557 2556 scope.go:117] "RemoveContainer" containerID="9fe349b22ada60d0771f751c741bba5e5efdf18c74a264d0b0d09ced77db9dab" Sep 08 17:20:26.055736 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.055671 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:20:26.197781 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.197748 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.197781 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.197789 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.197889 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198006 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.197945 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198005 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztjzt\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198042 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198059 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198083 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198125 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy\") pod \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\" (UID: \"076f4b03-485d-4b89-bcf9-c6aebdcdea83\") " Sep 08 17:20:26.198405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198161 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:20:26.198405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198195 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:26.198405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198222 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data" (OuterVolumeSpecName: "istio-data") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:26.198405 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198377 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198516 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198574 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198592 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-credential-socket\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198605 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istiod-ca-cert\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198614 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-workload-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.198646 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.198625 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-data\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.200263 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.200227 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token" (OuterVolumeSpecName: "istio-token") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:26.200420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.200309 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 17:20:26.200420 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.200337 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt" (OuterVolumeSpecName: "kube-api-access-ztjzt") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "kube-api-access-ztjzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:20:26.200520 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.200497 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "076f4b03-485d-4b89-bcf9-c6aebdcdea83" (UID: "076f4b03-485d-4b89-bcf9-c6aebdcdea83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 17:20:26.299291 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.299257 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-token\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.299291 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.299286 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-envoy\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.299291 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.299295 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076f4b03-485d-4b89-bcf9-c6aebdcdea83-istio-podinfo\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.299291 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.299305 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjzt\" (UniqueName: \"kubernetes.io/projected/076f4b03-485d-4b89-bcf9-c6aebdcdea83-kube-api-access-ztjzt\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:20:26.978822 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.978791 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" Sep 08 17:20:26.979258 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.978791 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x" event={"ID":"076f4b03-485d-4b89-bcf9-c6aebdcdea83","Type":"ContainerDied","Data":"08ede787b8fa71ed16b61a2092747bcfb599614d897c16490a56afa1e81a6fb3"} Sep 08 17:20:26.979258 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:26.978926 2556 scope.go:117] "RemoveContainer" containerID="a280a45534628d100cc51f96ba9cf664a8576f91313cd4eea58f6b63fa4e1f0d" Sep 08 17:20:27.000837 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:27.000811 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:20:27.006159 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:27.006127 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-338600-openshift-default-86548d45b6-tjj9x"] Sep 08 17:20:27.342466 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:20:27.342384 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" path="/var/lib/kubelet/pods/076f4b03-485d-4b89-bcf9-c6aebdcdea83/volumes" Sep 08 17:21:09.729879 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:21:09.729850 2556 scope.go:117] "RemoveContainer" containerID="df2405907f9dd00cd9c2888075308f8fa7c79e45f7fa56fc2438c49bbb1f3977" Sep 08 17:21:09.738229 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:21:09.738210 2556 scope.go:117] "RemoveContainer" containerID="a2b3bef0a4f11df863a63d835d96cad464127dcc13ea1da2ebd9312f1d5e947f" Sep 08 17:21:09.745802 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:21:09.745782 2556 scope.go:117] "RemoveContainer" containerID="c57dac020985f75b96f2ffa19d700da7726971347101150de4863883743bc570" Sep 08 17:21:49.824099 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:21:49.824063 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:22:14.523508 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:22:14.523466 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:22:22.410502 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:22:22.410461 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:22:29.517785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:22:29.517750 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:22:36.815805 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:22:36.815724 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:22:55.240940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:22:55.240901 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:23:03.418314 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:23:03.418280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:24:34.825639 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:24:34.825604 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:24:40.703935 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:24:40.703898 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:24:50.006800 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:24:50.006763 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:24:59.835866 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:24:59.835830 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:25:12.303036 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:25:12.302998 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:25:39.203169 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:25:39.203137 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:25:48.305940 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:25:48.305905 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:25:55.315727 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:25:55.315684 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:02.117136 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.117100 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:02.499808 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.499711 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:02.500187 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500166 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70633eae-2719-4899-a1f8-d232c0784dce" containerName="revoke-keys" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500190 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="70633eae-2719-4899-a1f8-d232c0784dce" containerName="revoke-keys" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500204 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500211 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500233 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500243 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500267 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.500280 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500275 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.500668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500381 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="70633eae-2719-4899-a1f8-d232c0784dce" containerName="revoke-keys" Sep 08 17:26:02.500668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500392 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:02.500668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500406 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.500668 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.500417 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="076f4b03-485d-4b89-bcf9-c6aebdcdea83" containerName="istio-proxy" Sep 08 17:26:02.503670 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.503647 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.506411 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.506389 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 17:26:02.507648 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.507628 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 17:26:02.507756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.507687 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 17:26:02.507756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.507725 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 17:26:02.507756 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.507692 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qpc2m\"" Sep 08 17:26:02.512893 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.512872 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:02.622769 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.622732 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.622975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.622815 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.622975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.622865 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.622975 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.622883 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.724206 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.724164 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.724450 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.724237 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.724450 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.724257 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.724450 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.724281 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.724984 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.724960 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.726769 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.726742 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.726856 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.726794 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.732017 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.731986 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq\") pod \"maas-api-dd87ddf4c-5jqsg\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:02.815962 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:02.815862 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:03.086215 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:03.086191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:03.087983 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:26:03.087957 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19ce79e_89fb_42c9_87b4_74d3813acda8.slice/crio-b0b161dfd14d4cbac3f40e760b989316a2df214f9679631f3fc3fc8e0c3b083a WatchSource:0}: Error finding container b0b161dfd14d4cbac3f40e760b989316a2df214f9679631f3fc3fc8e0c3b083a: Status 404 returned error can't find the container with id b0b161dfd14d4cbac3f40e760b989316a2df214f9679631f3fc3fc8e0c3b083a Sep 08 17:26:03.089449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:03.089428 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 17:26:03.656624 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:03.656591 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" event={"ID":"d19ce79e-89fb-42c9-87b4-74d3813acda8","Type":"ContainerStarted","Data":"b0b161dfd14d4cbac3f40e760b989316a2df214f9679631f3fc3fc8e0c3b083a"} Sep 08 17:26:05.668142 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:05.668097 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" event={"ID":"d19ce79e-89fb-42c9-87b4-74d3813acda8","Type":"ContainerStarted","Data":"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740"} Sep 08 17:26:05.668539 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:05.668158 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:05.688537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:05.688493 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" podStartSLOduration=2.020741977 podStartE2EDuration="3.688478325s" totalImagesPullingTime="1.667736348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:26:05.686643766 +0000 UTC m=+2096.831414329" watchObservedRunningTime="2026-09-08 17:26:05.688478325 +0000 UTC m=+2096.833248888" Sep 08 17:26:09.810376 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:09.810317 2556 scope.go:117] "RemoveContainer" containerID="dbd8b1f648ce18eb43829e18a57a491bbcf28b7c51f62f3f4caa4c16be1d2e17" Sep 08 17:26:09.818444 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:09.818425 2556 scope.go:117] "RemoveContainer" containerID="88805043720b2ee8a5a4fe873fc0e137566087468e6f3642947449d3ec744bcc" Sep 08 17:26:09.825514 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:09.825498 2556 scope.go:117] "RemoveContainer" containerID="69120462bd44ecfad988027ff9d23ce66c27595596cf11613196a79def50cfdf" Sep 08 17:26:09.920916 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:09.920879 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:10.217217 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.217185 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:10.217418 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.217373 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" podUID="d19ce79e-89fb-42c9-87b4-74d3813acda8" containerName="maas-api" containerID="cri-o://0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740" gracePeriod=30 Sep 08 17:26:10.222425 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.222400 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:10.246090 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.246064 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq"] Sep 08 17:26:10.246489 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.246474 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:10.246550 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.246491 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:10.246587 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.246551 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd354ff-1818-430c-9abd-0d4135f952ff" containerName="istio-proxy" Sep 08 17:26:10.249528 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.249513 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.259544 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.259515 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq"] Sep 08 17:26:10.398929 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.398901 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-metrics-certs\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.399067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.398953 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-maas-api-tls\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.399067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.399013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xhv\" (UniqueName: \"kubernetes.io/projected/71718ec7-e20c-4073-be55-4bdc48c7d3a7-kube-api-access-f8xhv\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.399067 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.399042 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71718ec7-e20c-4073-be55-4bdc48c7d3a7-openshift-service-ca\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.464641 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.464618 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:10.500089 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.499999 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-metrics-certs\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.500089 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.500078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-maas-api-tls\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.500348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.500197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xhv\" (UniqueName: \"kubernetes.io/projected/71718ec7-e20c-4073-be55-4bdc48c7d3a7-kube-api-access-f8xhv\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.500348 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.500235 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71718ec7-e20c-4073-be55-4bdc48c7d3a7-openshift-service-ca\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.501080 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.501055 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71718ec7-e20c-4073-be55-4bdc48c7d3a7-openshift-service-ca\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.502679 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.502660 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-maas-api-tls\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.502914 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.502892 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71718ec7-e20c-4073-be55-4bdc48c7d3a7-metrics-certs\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.508152 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.508123 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xhv\" (UniqueName: \"kubernetes.io/projected/71718ec7-e20c-4073-be55-4bdc48c7d3a7-kube-api-access-f8xhv\") pod \"maas-api-79cd597bc5-gmmjq\" (UID: \"71718ec7-e20c-4073-be55-4bdc48c7d3a7\") " pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.601205 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.601174 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs\") pod \"d19ce79e-89fb-42c9-87b4-74d3813acda8\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " Sep 08 17:26:10.601205 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.601210 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls\") pod \"d19ce79e-89fb-42c9-87b4-74d3813acda8\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " Sep 08 17:26:10.601504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.601249 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq\") pod \"d19ce79e-89fb-42c9-87b4-74d3813acda8\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " Sep 08 17:26:10.601504 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.601413 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca\") pod \"d19ce79e-89fb-42c9-87b4-74d3813acda8\" (UID: \"d19ce79e-89fb-42c9-87b4-74d3813acda8\") " Sep 08 17:26:10.601780 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.601753 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d19ce79e-89fb-42c9-87b4-74d3813acda8" (UID: "d19ce79e-89fb-42c9-87b4-74d3813acda8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 17:26:10.603592 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.603562 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d19ce79e-89fb-42c9-87b4-74d3813acda8" (UID: "d19ce79e-89fb-42c9-87b4-74d3813acda8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:26:10.603704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.603596 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d19ce79e-89fb-42c9-87b4-74d3813acda8" (UID: "d19ce79e-89fb-42c9-87b4-74d3813acda8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 17:26:10.603704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.603665 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq" (OuterVolumeSpecName: "kube-api-access-ncvbq") pod "d19ce79e-89fb-42c9-87b4-74d3813acda8" (UID: "d19ce79e-89fb-42c9-87b4-74d3813acda8"). InnerVolumeSpecName "kube-api-access-ncvbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 17:26:10.620744 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.620707 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:10.693283 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.693251 2556 generic.go:356] "Generic (PLEG): container finished" podID="d19ce79e-89fb-42c9-87b4-74d3813acda8" containerID="0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740" exitCode=0 Sep 08 17:26:10.693485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.693340 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" Sep 08 17:26:10.693485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.693418 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" event={"ID":"d19ce79e-89fb-42c9-87b4-74d3813acda8","Type":"ContainerDied","Data":"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740"} Sep 08 17:26:10.693589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.693499 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg" event={"ID":"d19ce79e-89fb-42c9-87b4-74d3813acda8","Type":"ContainerDied","Data":"b0b161dfd14d4cbac3f40e760b989316a2df214f9679631f3fc3fc8e0c3b083a"} Sep 08 17:26:10.693589 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.693540 2556 scope.go:117] "RemoveContainer" containerID="0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740" Sep 08 17:26:10.702751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.702727 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d19ce79e-89fb-42c9-87b4-74d3813acda8-openshift-service-ca\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:26:10.702751 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.702751 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-metrics-certs\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:26:10.702949 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.702762 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d19ce79e-89fb-42c9-87b4-74d3813acda8-maas-api-tls\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:26:10.702949 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.702770 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvbq\" (UniqueName: \"kubernetes.io/projected/d19ce79e-89fb-42c9-87b4-74d3813acda8-kube-api-access-ncvbq\") on node \"ip-10-0-132-198.ec2.internal\" DevicePath \"\"" Sep 08 17:26:10.706651 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.706581 2556 scope.go:117] "RemoveContainer" containerID="0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740" Sep 08 17:26:10.707007 ip-10-0-132-198 kubenswrapper[2556]: E0908 17:26:10.706982 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740\": container with ID starting with 0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740 not found: ID does not exist" containerID="0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740" Sep 08 17:26:10.707082 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.707020 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740"} err="failed to get container status \"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740\": rpc error: code = NotFound desc = could not find container \"0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740\": container with ID starting with 0ad3e562adae6f38f037273ef8c040870bda15f522c705eed6970c1b0a3f6740 not found: ID does not exist" Sep 08 17:26:10.720192 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.719704 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:10.723664 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.723636 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-5jqsg"] Sep 08 17:26:10.895966 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:10.895937 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq"] Sep 08 17:26:10.896955 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:26:10.896931 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71718ec7_e20c_4073_be55_4bdc48c7d3a7.slice/crio-a516efe2c3368d26cf7ed61ccf1a30bbef7af33611de99023080be24966cf04f WatchSource:0}: Error finding container a516efe2c3368d26cf7ed61ccf1a30bbef7af33611de99023080be24966cf04f: Status 404 returned error can't find the container with id a516efe2c3368d26cf7ed61ccf1a30bbef7af33611de99023080be24966cf04f Sep 08 17:26:11.342462 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:11.342427 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19ce79e-89fb-42c9-87b4-74d3813acda8" path="/var/lib/kubelet/pods/d19ce79e-89fb-42c9-87b4-74d3813acda8/volumes" Sep 08 17:26:11.700480 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:11.700443 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" event={"ID":"71718ec7-e20c-4073-be55-4bdc48c7d3a7","Type":"ContainerStarted","Data":"ff5c9c6c45981dc9d4832221a78f022a7a5be7ece7ae3a442462d898e6ce72a3"} Sep 08 17:26:11.700678 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:11.700499 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" event={"ID":"71718ec7-e20c-4073-be55-4bdc48c7d3a7","Type":"ContainerStarted","Data":"a516efe2c3368d26cf7ed61ccf1a30bbef7af33611de99023080be24966cf04f"} Sep 08 17:26:11.700678 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:11.700530 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:11.719610 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:11.719547 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" podStartSLOduration=1.419637462 podStartE2EDuration="1.719532599s" totalImagesPullingTime="299.895137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:26:11.717609669 +0000 UTC m=+2102.862380232" watchObservedRunningTime="2026-09-08 17:26:11.719532599 +0000 UTC m=+2102.864303159" Sep 08 17:26:16.701644 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:16.701606 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:17.712073 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:17.712043 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79cd597bc5-gmmjq" Sep 08 17:26:22.710222 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:22.710188 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:29.398980 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:29.398933 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-4wmfx"] Sep 08 17:26:38.419950 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:38.419918 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-wmfwk_74f6407b-e10f-4bfc-8217-79cdf3706719/copy-manifests/0.log" Sep 08 17:26:38.426725 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:38.426698 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-wmfwk_74f6407b-e10f-4bfc-8217-79cdf3706719/manager/0.log" Sep 08 17:26:38.763711 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:38.763622 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79cd597bc5-gmmjq_71718ec7-e20c-4073-be55-4bdc48c7d3a7/maas-api/0.log" Sep 08 17:26:40.526809 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:40.526778 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4wmfx_51772def-5c84-49be-ba3c-8498a8fea26d/limitador/0.log" Sep 08 17:26:40.984865 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:40.984835 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7_c18f8aa4-b156-48a3-ac6b-87405aaad7c6/istio-proxy/0.log" Sep 08 17:26:41.096945 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:41.096913 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-njtwm_c1dda4eb-600b-4b7e-ba4c-546ae562fff5/discovery/0.log" Sep 08 17:26:41.442949 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:41.442915 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/1.log" Sep 08 17:26:42.501622 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:42.501584 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wkbhl_79074beb-51f3-4b8d-8c7d-675a08ef8a34/main/0.log" Sep 08 17:26:50.994541 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:50.994500 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zmtz9_9853d638-cf47-4f8f-b45f-0a83f02af5d1/global-pull-secret-syncer/0.log" Sep 08 17:26:51.092195 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:51.092164 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gf6wq_71439f0e-1b37-4b69-9473-973fddc608a5/konnectivity-agent/0.log" Sep 08 17:26:51.147485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:51.147433 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-198.ec2.internal_56dc1b1e4db5094fc00daf3962293d7e/haproxy/0.log" Sep 08 17:26:56.031081 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:56.031051 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-4wmfx_51772def-5c84-49be-ba3c-8498a8fea26d/limitador/0.log" Sep 08 17:26:57.412160 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.412129 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/alertmanager/0.log" Sep 08 17:26:57.434669 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.434640 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/config-reloader/0.log" Sep 08 17:26:57.454704 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.454680 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/kube-rbac-proxy-web/0.log" Sep 08 17:26:57.473838 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.473811 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/kube-rbac-proxy/0.log" Sep 08 17:26:57.498441 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.498416 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/kube-rbac-proxy-metric/0.log" Sep 08 17:26:57.517397 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.517372 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/prom-label-proxy/0.log" Sep 08 17:26:57.549785 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.549755 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6573f9a7-a191-4457-89a8-3e93a99d27a5/init-config-reloader/0.log" Sep 08 17:26:57.617860 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.617831 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-9mdjs_7514196b-f24b-4e94-a455-379adc691f79/kube-state-metrics/0.log" Sep 08 17:26:57.635886 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.635861 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-9mdjs_7514196b-f24b-4e94-a455-379adc691f79/kube-rbac-proxy-main/0.log" Sep 08 17:26:57.657163 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.657134 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-9mdjs_7514196b-f24b-4e94-a455-379adc691f79/kube-rbac-proxy-self/0.log" Sep 08 17:26:57.686547 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.686518 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bf45677b9-6lzw4_77dc8729-7ba0-4229-8dce-2c61a315c4ff/metrics-server/0.log" Sep 08 17:26:57.710162 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.710131 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-sp77t_3803fb5f-cd1a-42e9-8c3f-87e2b76f4844/monitoring-plugin/0.log" Sep 08 17:26:57.755282 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.755248 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2zjlf_6c466071-335d-4f93-92d8-2830b8f01b18/node-exporter/0.log" Sep 08 17:26:57.775883 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.775854 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2zjlf_6c466071-335d-4f93-92d8-2830b8f01b18/kube-rbac-proxy/0.log" Sep 08 17:26:57.793827 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.793802 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2zjlf_6c466071-335d-4f93-92d8-2830b8f01b18/init-textfile/0.log" Sep 08 17:26:57.969018 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:57.968989 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2p9fk_02d58084-9a2b-49c6-a79e-0d09c632d179/kube-rbac-proxy-main/0.log" Sep 08 17:26:58.005859 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.005829 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2p9fk_02d58084-9a2b-49c6-a79e-0d09c632d179/kube-rbac-proxy-self/0.log" Sep 08 17:26:58.036339 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.036289 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2p9fk_02d58084-9a2b-49c6-a79e-0d09c632d179/openshift-state-metrics/0.log" Sep 08 17:26:58.077127 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.077099 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/prometheus/0.log" Sep 08 17:26:58.096524 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.096499 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/config-reloader/0.log" Sep 08 17:26:58.120410 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.120382 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/thanos-sidecar/0.log" Sep 08 17:26:58.144474 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.144449 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/kube-rbac-proxy-web/0.log" Sep 08 17:26:58.167624 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.167551 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/kube-rbac-proxy/0.log" Sep 08 17:26:58.190877 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.190845 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/kube-rbac-proxy-thanos/0.log" Sep 08 17:26:58.213216 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.213191 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_74cb9b9e-8724-420b-9508-d3c2f775afc8/init-config-reloader/0.log" Sep 08 17:26:58.243999 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.243972 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-q5fvg_2540541d-0d48-46f1-acd5-2bf9a2506373/prometheus-operator/0.log" Sep 08 17:26:58.260633 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.260608 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-q5fvg_2540541d-0d48-46f1-acd5-2bf9a2506373/kube-rbac-proxy/0.log" Sep 08 17:26:58.315635 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.315591 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-799f5c6974-z9tbx_024b00b1-9b61-4fbd-888b-5b70d1980e60/telemeter-client/0.log" Sep 08 17:26:58.334335 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.334304 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-799f5c6974-z9tbx_024b00b1-9b61-4fbd-888b-5b70d1980e60/reload/0.log" Sep 08 17:26:58.353952 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.353922 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-799f5c6974-z9tbx_024b00b1-9b61-4fbd-888b-5b70d1980e60/kube-rbac-proxy/0.log" Sep 08 17:26:58.385608 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.385569 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/thanos-query/0.log" Sep 08 17:26:58.408882 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.408858 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/kube-rbac-proxy-web/0.log" Sep 08 17:26:58.431449 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.431376 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/kube-rbac-proxy/0.log" Sep 08 17:26:58.450498 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.450459 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/prom-label-proxy/0.log" Sep 08 17:26:58.468451 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.468426 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/kube-rbac-proxy-rules/0.log" Sep 08 17:26:58.489249 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:26:58.489224 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59f5fb6d78-6jmft_4abbe321-dac0-433b-b533-1b5afa2cc179/kube-rbac-proxy-metrics/0.log" Sep 08 17:27:01.158818 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.158782 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm"] Sep 08 17:27:01.159269 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.159249 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19ce79e-89fb-42c9-87b4-74d3813acda8" containerName="maas-api" Sep 08 17:27:01.159352 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.159272 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19ce79e-89fb-42c9-87b4-74d3813acda8" containerName="maas-api" Sep 08 17:27:01.159433 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.159418 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19ce79e-89fb-42c9-87b4-74d3813acda8" containerName="maas-api" Sep 08 17:27:01.162908 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.162889 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.165495 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.165473 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"openshift-service-ca.crt\"" Sep 08 17:27:01.165679 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.165512 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"kube-root-ca.crt\"" Sep 08 17:27:01.165844 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.165799 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7djdj\"/\"default-dockercfg-q2jfr\"" Sep 08 17:27:01.171959 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.171935 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm"] Sep 08 17:27:01.200556 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.200525 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-lib-modules\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.200726 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.200566 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-sys\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.200726 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.200587 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwsk\" (UniqueName: \"kubernetes.io/projected/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-kube-api-access-fbwsk\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.200726 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.200651 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-podres\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.200726 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.200690 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-proc\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301696 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301657 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-podres\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-proc\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301779 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-lib-modules\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301805 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-sys\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301832 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwsk\" (UniqueName: \"kubernetes.io/projected/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-kube-api-access-fbwsk\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301851 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-proc\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.301891 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301852 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-podres\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.302118 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301898 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-sys\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.302118 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.301923 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-lib-modules\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.309724 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.309701 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwsk\" (UniqueName: \"kubernetes.io/projected/d4cf1ce2-35ed-4cea-9f84-82c156c45cdb-kube-api-access-fbwsk\") pod \"perf-node-gather-daemonset-nc9bm\" (UID: \"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.473026 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.472936 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.741225 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.741199 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm"] Sep 08 17:27:01.743250 ip-10-0-132-198 kubenswrapper[2556]: W0908 17:27:01.743225 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4cf1ce2_35ed_4cea_9f84_82c156c45cdb.slice/crio-bba49164bf35fbb0df74d2c9cc69298b5b6d1c6b00cd6b5bc60f85683554cf30 WatchSource:0}: Error finding container bba49164bf35fbb0df74d2c9cc69298b5b6d1c6b00cd6b5bc60f85683554cf30: Status 404 returned error can't find the container with id bba49164bf35fbb0df74d2c9cc69298b5b6d1c6b00cd6b5bc60f85683554cf30 Sep 08 17:27:01.955156 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.955122 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" event={"ID":"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb","Type":"ContainerStarted","Data":"4d329b1176fae064642230aa06d069873f5277f3fc5676384c1eba4a97724d54"} Sep 08 17:27:01.955342 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.955268 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:01.955342 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.955304 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" event={"ID":"d4cf1ce2-35ed-4cea-9f84-82c156c45cdb","Type":"ContainerStarted","Data":"bba49164bf35fbb0df74d2c9cc69298b5b6d1c6b00cd6b5bc60f85683554cf30"} Sep 08 17:27:01.970485 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:01.970438 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" podStartSLOduration=0.970423709 podStartE2EDuration="970.423709ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:27:01.969567981 +0000 UTC m=+2153.114338544" watchObservedRunningTime="2026-09-08 17:27:01.970423709 +0000 UTC m=+2153.115194269" Sep 08 17:27:03.090642 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:03.090611 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xc9ml_2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3/dns/0.log" Sep 08 17:27:03.107700 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:03.107675 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xc9ml_2acadec1-8dde-4dea-bb5a-5eb7b2f6a6e3/kube-rbac-proxy/0.log" Sep 08 17:27:03.149256 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:03.149226 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nt2lq_08fd9286-b2e0-4f1e-afba-4a87fc36592b/dns-node-resolver/0.log" Sep 08 17:27:03.707260 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:03.707221 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22bfg_c6b8082d-9f7d-402c-a13c-04e9d22707ca/node-ca/0.log" Sep 08 17:27:04.620807 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:04.620775 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7_c18f8aa4-b156-48a3-ac6b-87405aaad7c6/istio-proxy/0.log" Sep 08 17:27:05.005301 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:05.005213 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-njtwm_c1dda4eb-600b-4b7e-ba4c-546ae562fff5/discovery/0.log" Sep 08 17:27:05.086261 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:05.086228 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/0.log" Sep 08 17:27:05.101969 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:05.101922 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/1.log" Sep 08 17:27:06.086710 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:06.086663 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xgx5g_bd0702a9-c059-4dcf-91ec-e204eada2bd9/serve-healthcheck-canary/0.log" Sep 08 17:27:06.556705 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:06.556676 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hw7h4_b7038167-6504-4a28-ba03-49fdb45ab0b4/kube-rbac-proxy/0.log" Sep 08 17:27:06.578150 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:06.578118 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hw7h4_b7038167-6504-4a28-ba03-49fdb45ab0b4/exporter/0.log" Sep 08 17:27:06.597862 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:06.597836 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hw7h4_b7038167-6504-4a28-ba03-49fdb45ab0b4/extractor/0.log" Sep 08 17:27:07.970066 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:07.970036 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-nc9bm" Sep 08 17:27:09.467797 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:09.467766 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-wmfwk_74f6407b-e10f-4bfc-8217-79cdf3706719/manager/0.log" Sep 08 17:27:09.485104 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:09.485075 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-wmfwk_74f6407b-e10f-4bfc-8217-79cdf3706719/copy-manifests/0.log" Sep 08 17:27:11.004569 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:11.004481 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74fc5c54f8-sx7nh_f850b741-e351-4a1d-8adf-f802652192ce/manager/0.log" Sep 08 17:27:11.599494 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:11.599449 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7_c18f8aa4-b156-48a3-ac6b-87405aaad7c6/istio-proxy/0.log" Sep 08 17:27:12.016031 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:12.015998 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-njtwm_c1dda4eb-600b-4b7e-ba4c-546ae562fff5/discovery/0.log" Sep 08 17:27:12.064032 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:12.063992 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/0.log" Sep 08 17:27:12.079189 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:12.079158 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/1.log" Sep 08 17:27:13.657567 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:13.657535 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6slf7_c18f8aa4-b156-48a3-ac6b-87405aaad7c6/istio-proxy/0.log" Sep 08 17:27:14.088660 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:14.088562 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-njtwm_c1dda4eb-600b-4b7e-ba4c-546ae562fff5/discovery/0.log" Sep 08 17:27:14.143264 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:14.143230 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/0.log" Sep 08 17:27:14.160924 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:14.160891 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h9929_deb2cf38-f822-4c65-a4f9-94eaac8469b6/istio-proxy/1.log" Sep 08 17:27:17.361951 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.361913 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6c526_94e7db0e-f57b-41b1-a217-9adbe1b83b4d/kube-multus/0.log" Sep 08 17:27:17.743028 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.743000 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/kube-multus-additional-cni-plugins/0.log" Sep 08 17:27:17.770752 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.770719 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/egress-router-binary-copy/0.log" Sep 08 17:27:17.788979 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.788952 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/cni-plugins/0.log" Sep 08 17:27:17.807601 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.807571 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/bond-cni-plugin/0.log" Sep 08 17:27:17.828463 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.828425 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/routeoverride-cni/0.log" Sep 08 17:27:17.848429 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.848399 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/whereabouts-cni-bincopy/0.log" Sep 08 17:27:17.867537 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.867508 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdpks_cfd00c50-72f9-4af9-964e-bdb82bb89119/whereabouts-cni/0.log" Sep 08 17:27:17.969378 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.969351 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j74bn_5acd11f2-9934-413b-ac7a-4646902745f6/network-metrics-daemon/0.log" Sep 08 17:27:17.985335 ip-10-0-132-198 kubenswrapper[2556]: I0908 17:27:17.985297 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j74bn_5acd11f2-9934-413b-ac7a-4646902745f6/kube-rbac-proxy/0.log"