Jul 26 15:05:24.698842 ip-10-0-139-15 systemd[1]: Starting Kubernetes Kubelet... Jul 26 15:05:25.060164 ip-10-0-139-15 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:25.060164 ip-10-0-139-15 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 15:05:25.060164 ip-10-0-139-15 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:25.060164 ip-10-0-139-15 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 15:05:25.060164 ip-10-0-139-15 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:05:25.060948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.060855 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 15:05:25.064447 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064426 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.064447 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064446 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064451 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064454 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064458 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064461 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064465 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064468 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064471 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064474 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064477 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064481 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064483 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064486 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064488 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064491 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064493 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064496 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064498 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064504 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064507 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.064534 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064509 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064512 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064528 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064531 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064534 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064537 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064540 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064543 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064546 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064549 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064551 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064554 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064557 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064559 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064562 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064566 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064570 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064573 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064576 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.065119 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064578 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064583 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064586 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064590 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064592 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064595 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064598 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064600 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064603 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064605 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064608 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064610 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064613 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064615 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064618 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064621 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064623 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064626 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064629 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.065628 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064631 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064634 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064637 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064640 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064642 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064645 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064648 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064650 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064653 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064656 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064658 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064661 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064663 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064666 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064669 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064671 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064674 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064676 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064679 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064681 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.066088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064684 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064687 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064690 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064692 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064694 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064699 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064702 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064705 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064709 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064712 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064715 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064718 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064720 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064723 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064726 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064729 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064731 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064734 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064737 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.066584 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.064739 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065463 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065469 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065473 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065475 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065478 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065481 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065483 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065486 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065488 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065491 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065495 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065500 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065503 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065507 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065510 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065513 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065532 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065535 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.067033 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065538 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065540 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065544 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065546 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065549 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065552 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065554 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065557 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065560 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065562 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065565 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065568 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065570 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065573 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065576 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065578 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065581 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065583 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065586 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065588 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.067498 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065591 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065594 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065597 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065599 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065602 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065605 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065607 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065610 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065613 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065616 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065618 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065621 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065623 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065626 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065630 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065633 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065635 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065638 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065641 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065643 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.068030 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065646 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065648 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065651 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065654 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065656 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065659 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065662 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065664 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065667 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065669 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065672 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065674 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065677 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065679 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065682 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065684 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065687 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065691 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065693 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065696 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.068543 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065699 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065702 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065705 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065709 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065713 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065716 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065719 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065721 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065724 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065727 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065729 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065733 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065735 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065738 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065740 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065743 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065745 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065748 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065751 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.069048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065753 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.065757 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066136 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066141 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066143 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066146 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066149 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066152 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066154 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066157 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066160 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066163 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066165 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066168 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066171 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066173 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066176 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066178 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.069502 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066181 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066185 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066188 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066191 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066194 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066197 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066200 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066203 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066205 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066208 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066212 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066216 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066219 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066222 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066224 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066227 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066229 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066232 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066235 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.069952 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066237 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066240 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066242 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066245 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066247 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066250 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066253 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066255 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066258 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066260 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066263 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066266 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066268 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066271 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066273 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066276 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066278 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066280 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066284 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066286 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066289 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.070415 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066291 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066294 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066296 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066299 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066301 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066303 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066306 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066308 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066311 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066313 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066318 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066320 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066323 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066325 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066328 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066330 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066333 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066336 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066338 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066341 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.070951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066343 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066346 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066349 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066351 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066354 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066356 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066359 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066361 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066364 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066366 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066368 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066372 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066374 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066377 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066379 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066383 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066386 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066389 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066392 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066394 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.071434 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066397 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066399 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.066403 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066481 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066489 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066496 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066501 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066505 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066509 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066514 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066533 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066536 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066539 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066542 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066545 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066549 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066552 2575 flags.go:64] FLAG: --cgroup-root="" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066555 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066558 2575 flags.go:64] FLAG: --client-ca-file="" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066561 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066564 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066568 2575 flags.go:64] FLAG: --cluster-domain="" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066571 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 15:05:25.071916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066574 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066577 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066580 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066584 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066587 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066590 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066593 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066596 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066599 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066602 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066605 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066609 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066613 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066616 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066618 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066621 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066624 2575 flags.go:64] FLAG: --enable-server="true" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066627 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066632 2575 flags.go:64] FLAG: --event-burst="100" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066635 2575 flags.go:64] FLAG: --event-qps="50" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066638 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066641 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066644 2575 flags.go:64] FLAG: --eviction-hard="" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066648 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 15:05:25.072500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066651 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066654 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066657 2575 flags.go:64] FLAG: --eviction-soft="" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066660 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066662 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066665 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066668 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066671 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066673 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066676 2575 flags.go:64] FLAG: --feature-gates="" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066680 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066682 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066685 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066688 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066691 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066694 2575 flags.go:64] FLAG: --help="false" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066697 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066699 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066702 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066705 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066709 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066712 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066715 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 15:05:25.073109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066718 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066721 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066724 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066727 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066730 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066733 2575 flags.go:64] FLAG: --kube-reserved="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066735 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066738 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066741 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066744 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066747 2575 flags.go:64] FLAG: --lock-file="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066750 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066753 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066756 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066761 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066764 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066767 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066769 2575 flags.go:64] FLAG: --logging-format="text" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066773 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066776 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066778 2575 flags.go:64] FLAG: --manifest-url="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066781 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066785 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066788 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066794 2575 flags.go:64] FLAG: --max-pods="110" Jul 26 15:05:25.073710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066796 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066799 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066802 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066805 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066808 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066811 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066814 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066821 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066824 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066827 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066830 2575 flags.go:64] FLAG: --pod-cidr="" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066833 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066838 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066841 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066844 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066847 2575 flags.go:64] FLAG: --port="10250" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066851 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066854 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d47fe25216273611" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066857 2575 flags.go:64] FLAG: --qos-reserved="" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066860 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066863 2575 flags.go:64] FLAG: --register-node="true" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066866 2575 flags.go:64] FLAG: --register-with-taints="" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066870 2575 flags.go:64] FLAG: --registry-burst="10" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066872 2575 flags.go:64] FLAG: --registry-qps="5" Jul 26 15:05:25.074313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066875 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066878 2575 flags.go:64] FLAG: --reserved-memory="" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066882 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066885 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066888 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066891 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066893 2575 flags.go:64] FLAG: --runonce="false" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066897 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066900 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066903 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066906 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066909 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066912 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066915 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066918 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066921 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066924 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066926 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066929 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066932 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066934 2575 flags.go:64] FLAG: --system-cgroups="" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066937 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066942 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066945 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066947 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 15:05:25.074956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066951 2575 flags.go:64] FLAG: --tls-min-version="" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066954 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066956 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066959 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066962 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066965 2575 flags.go:64] FLAG: --v="2" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066969 2575 flags.go:64] FLAG: --version="false" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066973 2575 flags.go:64] FLAG: --vmodule="" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066976 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.066979 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067096 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067099 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067102 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067104 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067109 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067111 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067114 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067117 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067119 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067123 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067126 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067128 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.075563 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067131 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067134 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067136 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067139 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067142 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067145 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067147 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067150 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067152 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067155 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067157 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067162 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067165 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067168 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067172 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067175 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067177 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067180 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067183 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.076104 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067185 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067188 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067190 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067193 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067195 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067199 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067201 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067204 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067206 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067208 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067212 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067215 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067217 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067220 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067222 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067225 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067227 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067230 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067232 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.076583 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067235 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067238 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067240 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067242 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067245 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067248 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067250 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067252 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067255 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067258 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067260 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067262 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067265 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067267 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067270 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067272 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067275 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067277 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067282 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.077088 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067286 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067289 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067291 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067294 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067298 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067301 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067303 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067306 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067308 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067311 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067313 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067316 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067318 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067321 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067323 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067326 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067330 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067333 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067336 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.077575 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067339 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067342 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067345 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067348 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067351 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067354 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067356 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067359 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067362 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067364 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.067367 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.067377 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.073878 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.073894 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.073915 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 15:05:25.078048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.073924 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073980 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073986 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073990 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073993 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073996 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.073999 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074002 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074004 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074007 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074010 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074012 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074015 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074019 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074023 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074026 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074029 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074031 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074034 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.078413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074037 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074039 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074042 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074044 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074047 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074050 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074053 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074055 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074058 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074060 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074063 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074067 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074070 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074073 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074076 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074078 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074081 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074083 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074086 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074088 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.078901 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074091 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074093 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074096 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074098 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074100 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074103 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074105 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074108 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074110 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074113 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074115 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074118 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074120 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074123 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074125 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074128 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074130 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074133 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074135 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074137 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.079374 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074140 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074143 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074145 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074148 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074152 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074154 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074157 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074159 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074162 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074164 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074167 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074169 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074173 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074176 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074179 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074182 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074184 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074187 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074190 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.079864 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074192 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074195 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074197 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074200 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074203 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074205 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074207 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074210 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074212 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074215 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074217 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074220 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074222 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074224 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074227 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074230 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074232 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074235 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074238 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074241 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.080325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074243 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074247 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.074252 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074367 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074372 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074376 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074378 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074381 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074384 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074386 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074389 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074391 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074394 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074397 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074400 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:05:25.080867 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074402 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074405 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074407 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074409 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074412 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074414 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074417 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074420 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074422 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074425 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074429 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074432 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074434 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074437 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074439 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074442 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074445 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074447 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074450 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:05:25.081268 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074452 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074454 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074457 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074459 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074462 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074464 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074468 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074471 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074473 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074476 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074478 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074481 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074483 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074486 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074488 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074490 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074493 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074495 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074498 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:05:25.081753 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074500 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074503 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074505 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074507 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074510 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074512 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074535 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074539 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074542 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074545 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074548 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074551 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074554 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074556 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074559 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074561 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074564 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074566 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074568 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074571 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:05:25.082252 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074573 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074576 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074578 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074580 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074583 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074586 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074588 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074591 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074593 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074595 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074601 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074603 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074606 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074608 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074611 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074613 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074616 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074618 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074621 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074623 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:05:25.082877 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074626 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074628 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074631 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074634 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074638 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074641 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074644 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074647 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:25.074650 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.074655 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.074771 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.077617 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.078850 2575 server.go:1019] "Starting client certificate rotation" Jul 26 15:05:25.083368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.078951 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:05:25.083713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.078981 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:05:25.083713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.081129 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:05:25.083713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.082303 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:05:25.092443 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.092420 2575 log.go:25] "Validated CRI v1 runtime API" Jul 26 15:05:25.094528 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.094497 2575 log.go:25] "Validated CRI v1 image API" Jul 26 15:05:25.096050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.096031 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 15:05:25.098529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.098492 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 813e03e8-3ba7-40a1-a0c0-6479a42d823d:/dev/nvme0n1p4 c7290e7e-4821-4f18-8aa5-3ba96dbe149b:/dev/nvme0n1p3] Jul 26 15:05:25.098604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.098513 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 15:05:25.105183 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.105047 2575 manager.go:216] Machine: {Timestamp:2026-07-26 15:05:25.102170382 +0000 UTC m=+0.293601457 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099897 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac130ffa856e118d905e108dc04e6 SystemUUID:ec2ac130-ffa8-56e1-18d9-05e108dc04e6 BootID:c068084a-bfbd-46a5-b9f8-25429c68d02c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:dd:a8:05:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:dd:a8:05:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:08:16:4d:43:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 15:05:25.105183 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.105165 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 15:05:25.105307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.105254 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 15:05:25.107175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.107158 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:05:25.107914 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.107884 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 15:05:25.108059 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.107918 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 15:05:25.109258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.109246 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 15:05:25.109290 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.109262 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 15:05:25.109290 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.109270 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:05:25.109930 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.109919 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:05:25.110779 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.110768 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 15:05:25.112679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112667 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:05:25.112804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112795 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 15:05:25.112842 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112827 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 26 15:05:25.112883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112842 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 15:05:25.112883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112867 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 15:05:25.112883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112882 2575 kubelet.go:398] "Adding apiserver pod source" Jul 26 15:05:25.112997 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.112894 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 15:05:25.113220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.113209 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:05:25.113272 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.113258 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:05:25.115194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.115177 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 15:05:25.117121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.117108 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 15:05:25.117185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.117139 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 15:05:25.118608 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118596 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 15:05:25.118647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118612 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 15:05:25.118647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118619 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 15:05:25.118647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118638 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118650 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118658 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118669 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118676 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118683 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118690 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118715 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 15:05:25.118746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.118720 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 15:05:25.119444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.119433 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 15:05:25.119472 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.119445 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 15:05:25.122876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.122849 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 15:05:25.122876 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.122866 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 15:05:25.123035 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.122907 2575 server.go:1268] "Started kubelet" Jul 26 15:05:25.123035 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.122930 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 15:05:25.123035 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.122984 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 15:05:25.123406 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.123341 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 15:05:25.123502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.123434 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 15:05:25.124097 ip-10-0-139-15 systemd[1]: Started Kubernetes Kubelet. Jul 26 15:05:25.126195 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.125131 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 15:05:25.126195 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.125923 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 26 15:05:25.126797 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.126772 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2lnqg" Jul 26 15:05:25.129558 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.129537 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 15:05:25.131105 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.130114 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-15.ec2.internal.18c5df476932ea3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-15.ec2.internal,UID:ip-10-0-139-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-15.ec2.internal,},FirstTimestamp:2026-07-26 15:05:25.122886206 +0000 UTC m=+0.314317287,LastTimestamp:2026-07-26 15:05:25.122886206 +0000 UTC m=+0.314317287,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-15.ec2.internal,}" Jul 26 15:05:25.132918 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.132901 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 15:05:25.132918 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.132911 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 15:05:25.133066 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.133009 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.133507 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133486 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2lnqg" Jul 26 15:05:25.133507 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133505 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 15:05:25.133786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133762 2575 factory.go:55] Registering systemd factory Jul 26 15:05:25.133876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133790 2575 factory.go:223] Registration of the systemd container factory successfully Jul 26 15:05:25.133876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133770 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 15:05:25.133876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133836 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 15:05:25.134040 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133950 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 26 15:05:25.134040 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.133960 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 26 15:05:25.134040 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134041 2575 factory.go:153] Registering CRI-O factory Jul 26 15:05:25.134150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134052 2575 factory.go:223] Registration of the crio container factory successfully Jul 26 15:05:25.134150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134102 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 15:05:25.134150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134127 2575 factory.go:103] Registering Raw factory Jul 26 15:05:25.134150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134142 2575 manager.go:1196] Started watching for new ooms in manager Jul 26 15:05:25.134535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.134504 2575 manager.go:349] Starting recovery of all containers Jul 26 15:05:25.142785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.142763 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:25.143142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.143127 2575 manager.go:354] Recovery completed Jul 26 15:05:25.145629 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.145595 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-15.ec2.internal\" not found" node="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.148172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.148154 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.150359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150264 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.150359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150293 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.150359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150308 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.150888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150871 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 15:05:25.150888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150886 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 15:05:25.151030 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150894 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.151030 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.150902 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:05:25.151699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.151680 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.151777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.151712 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.151777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.151728 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.153025 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.153013 2575 policy_none.go:49] "None policy: Start" Jul 26 15:05:25.153068 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.153030 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 15:05:25.153068 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.153040 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 15:05:25.154080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.154069 2575 policy_none.go:47] "Start" Jul 26 15:05:25.193134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193113 2575 manager.go:337] "Starting Device Plugin manager" Jul 26 15:05:25.193261 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.193153 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 15:05:25.193261 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193167 2575 server.go:87] "Starting device plugin registration server" Jul 26 15:05:25.193446 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193435 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 15:05:25.193483 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193448 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 15:05:25.193597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193570 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 15:05:25.193723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193666 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 15:05:25.193723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.193676 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 15:05:25.194376 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.194357 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 15:05:25.194449 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.194401 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.252485 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.252451 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 15:05:25.253943 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.253922 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 15:05:25.254047 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.253957 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 15:05:25.254047 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.253978 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 15:05:25.254047 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.254020 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 15:05:25.256207 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.256188 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:25.294225 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.294187 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.295207 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.295188 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.295327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.295224 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.295327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.295239 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.295327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.295266 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.301311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.301295 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.301396 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.301320 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-15.ec2.internal\": node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.313667 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.313607 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.354145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.354112 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal"] Jul 26 15:05:25.354258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.354196 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.355157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.355139 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.355247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.355173 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.355247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.355187 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.356411 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.356395 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.356505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.356490 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.356578 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.356532 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.357086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357071 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.357086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357080 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.357222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357099 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.357222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357105 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.357222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357114 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.357222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.357119 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.358143 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.358126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.358231 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.358149 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:05:25.358801 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.358787 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:05:25.358868 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.358808 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:05:25.358868 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.358820 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:05:25.373086 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.373066 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-15.ec2.internal\" not found" node="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.376990 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.376976 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-15.ec2.internal\" not found" node="ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.413704 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.413685 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.435497 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.435477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.435601 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.435503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.435601 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.435544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9f78f483deac2095511863fac8041de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-15.ec2.internal\" (UID: \"a9f78f483deac2095511863fac8041de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.513976 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.513940 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.536357 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9f78f483deac2095511863fac8041de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-15.ec2.internal\" (UID: \"a9f78f483deac2095511863fac8041de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.536463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.536463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.536463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.536463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9f78f483deac2095511863fac8041de-config\") pod \"kube-apiserver-proxy-ip-10-0-139-15.ec2.internal\" (UID: \"a9f78f483deac2095511863fac8041de\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.536634 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.536449 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b6b5cf3c3dabcc7864a976f54f07afe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal\" (UID: \"0b6b5cf3c3dabcc7864a976f54f07afe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.614735 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.614671 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.675160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.675136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.680063 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:25.680046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:25.714927 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.714898 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:25.815405 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:25.815370 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:27.209701 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.209617 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:27.310140 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.310104 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-15.ec2.internal\" not found" Jul 26 15:05:27.329744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.329723 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:27.372726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.372694 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 15:05:27.373390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.372853 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:05:27.373390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.372876 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:05:27.373390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.372874 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:05:27.407058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.407007 2575 apiserver.go:52] "Watching apiserver" Jul 26 15:05:27.426905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.426874 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" Jul 26 15:05:27.426905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.426897 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 15:05:27.427865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.427845 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 15:05:27.428188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.428170 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-27mk5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56","openshift-image-registry/node-ca-r7sls","openshift-multus/multus-fls4b","openshift-network-diagnostics/network-check-target-j6cxn","openshift-network-operator/iptables-alerter-65bdl","kube-system/global-pull-secret-syncer-k44rb","kube-system/konnectivity-agent-xvmbh","openshift-cluster-node-tuning-operator/tuned-rch2d","openshift-dns/node-resolver-lpdkp","openshift-multus/multus-additional-cni-plugins-54ff4","openshift-multus/network-metrics-daemon-5ts4r"] Jul 26 15:05:27.430071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.430051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.431181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.431161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.431392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.431361 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 15:00:26 +0000 UTC" deadline="2028-02-22 09:49:40.602148841 +0000 UTC" Jul 26 15:05:27.431392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.431390 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13818h44m13.17076292s" Jul 26 15:05:27.434500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.433712 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.434612 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.434580 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 15:05:27.435296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.434843 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.435296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.434988 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 15:05:27.435296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.435043 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.435502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.435349 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 15:05:27.435502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.435383 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 15:05:27.435597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.435547 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.435597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.435549 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 15:05:27.437078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.437165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437153 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.437439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437338 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.437573 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437557 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:05:27.437644 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437585 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 15:05:27.437644 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437619 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kcrhs\"" Jul 26 15:05:27.437737 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.437671 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" Jul 26 15:05:27.438355 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.438321 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p5tnt\"" Jul 26 15:05:27.438438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.438428 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-knwdb\"" Jul 26 15:05:27.439000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.438981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:27.439085 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.439066 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:27.439447 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.439425 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.439516 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.439470 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 15:05:27.439808 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.439793 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 15:05:27.440324 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.440269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.440749 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.440732 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.441044 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.441028 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bbt8m\"" Jul 26 15:05:27.441106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.441061 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.441106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.441076 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:05:27.441610 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.441589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.441712 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.441694 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:27.442576 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.442561 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.442757 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.442744 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 15:05:27.442828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.442775 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.442882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.442870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tcfgc\"" Jul 26 15:05:27.443001 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.442987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.444098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.444084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.445300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.445119 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.445437 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.445394 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f78f483deac2095511863fac8041de.slice/crio-7af3c81de1c78319beb3c46a016fa7e9d39e1a909537f03bf483da09343c1461 WatchSource:0}: Error finding container 7af3c81de1c78319beb3c46a016fa7e9d39e1a909537f03bf483da09343c1461: Status 404 returned error can't find the container with id 7af3c81de1c78319beb3c46a016fa7e9d39e1a909537f03bf483da09343c1461 Jul 26 15:05:27.445675 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.445652 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 15:05:27.445757 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.445688 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 15:05:27.445757 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.445744 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6b5cf3c3dabcc7864a976f54f07afe.slice/crio-bb9c546b913027a49642fee393c9820b186e018e871e99cb00372983679fe84b WatchSource:0}: Error finding container bb9c546b913027a49642fee393c9820b186e018e871e99cb00372983679fe84b: Status 404 returned error can't find the container with id bb9c546b913027a49642fee393c9820b186e018e871e99cb00372983679fe84b Jul 26 15:05:27.445867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.445755 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vml2n\"" Jul 26 15:05:27.446495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.446469 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.446495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.446484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xb6gx\"" Jul 26 15:05:27.446683 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.446623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.446966 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.446946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.447037 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447024 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:05:27.447350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447335 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 15:05:27.447656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447633 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 15:05:27.447730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l8lwl\"" Jul 26 15:05:27.447730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447696 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal"] Jul 26 15:05:27.447861 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447732 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal"] Jul 26 15:05:27.447861 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.447818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.448008 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.447873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:27.449039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.449024 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 15:05:27.449142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.449053 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 15:05:27.449142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.449087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6z48m\"" Jul 26 15:05:27.450066 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.450043 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:05:27.457647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.457630 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljsd5" Jul 26 15:05:27.465466 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.465413 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ljsd5" Jul 26 15:05:27.528627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.528601 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 15:05:27.532804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39de062-6cbe-41fe-82e7-6133b46a5bca-host\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.532804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-modprobe-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-device-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-kubelet\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-k8s-cni-cncf-io\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtht\" (UniqueName: \"kubernetes.io/projected/fffe4393-063b-4937-97f4-35ace2c6f1d6-kube-api-access-8dtht\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-os-release\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb496619-1f24-4fba-a0d2-cb1f942047df-konnectivity-ca\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-slash\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.532958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-node-log\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-netd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.532986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-cni-binary-copy\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-netns\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-hostroot\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-daemon-config\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-tmp\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-system-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c08065a2-c657-476e-b036-7ce61e845062-host-slash\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-systemd\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-var-lib-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-run\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cnibin\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jn55\" (UniqueName: \"kubernetes.io/projected/f137a282-0be2-47f7-aa1c-b2c5edae3249-kube-api-access-5jn55\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-sys\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfsd\" (UniqueName: \"kubernetes.io/projected/c08065a2-c657-476e-b036-7ce61e845062-kube-api-access-qhfsd\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-kubernetes\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-lib-modules\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-var-lib-kubelet\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-kubelet-config\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-netns\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-env-overrides\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-script-lib\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-etc-kubernetes\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.533584 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffe4393-063b-4937-97f4-35ace2c6f1d6-tmp-dir\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c08065a2-c657-476e-b036-7ce61e845062-iptables-alerter-script\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysconfig\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-registration-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-systemd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vml\" (UniqueName: \"kubernetes.io/projected/96797814-df34-4f04-a4b0-8c054e5512d1-kube-api-access-82vml\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-sys-fs\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-tuned\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66x8m\" (UniqueName: \"kubernetes.io/projected/5dd4f240-48ff-4c88-b6c4-d219767ed76c-kube-api-access-66x8m\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-host\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cs5\" (UniqueName: \"kubernetes.io/projected/b39de062-6cbe-41fe-82e7-6133b46a5bca-kube-api-access-m2cs5\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb496619-1f24-4fba-a0d2-cb1f942047df-agent-certs\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckm9\" (UniqueName: \"kubernetes.io/projected/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-kube-api-access-9ckm9\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-systemd-units\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.534033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvscw\" (UniqueName: \"kubernetes.io/projected/964bdece-796d-4db9-b435-0a59fa03921b-kube-api-access-gvscw\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-cnibin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-socket-dir-parent\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.533974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-bin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-kubelet\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-bin\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-multus\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-conf\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-binary-copy\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-dbus\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-socket-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-os-release\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-conf-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.534719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vl5\" (UniqueName: \"kubernetes.io/projected/144b6fd1-04ca-4c47-9093-ae50d3a088e0-kube-api-access-84vl5\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39de062-6cbe-41fe-82e7-6133b46a5bca-serviceca\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-ovn\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-log-socket\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96797814-df34-4f04-a4b0-8c054e5512d1-ovn-node-metrics-cert\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-multus-certs\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffe4393-063b-4937-97f4-35ace2c6f1d6-hosts-file\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-system-cni-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-etc-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-config\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.535194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.534592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.551039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.550988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" event={"ID":"0b6b5cf3c3dabcc7864a976f54f07afe","Type":"ContainerStarted","Data":"bb9c546b913027a49642fee393c9820b186e018e871e99cb00372983679fe84b"} Jul 26 15:05:27.552027 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.552008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" event={"ID":"a9f78f483deac2095511863fac8041de","Type":"ContainerStarted","Data":"7af3c81de1c78319beb3c46a016fa7e9d39e1a909537f03bf483da09343c1461"} Jul 26 15:05:27.635176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-node-log\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-netd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-cni-binary-copy\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-netns\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-hostroot\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-daemon-config\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-node-log\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-tmp\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-netd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-netns\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-hostroot\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-system-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-system-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c08065a2-c657-476e-b036-7ce61e845062-host-slash\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c08065a2-c657-476e-b036-7ce61e845062-host-slash\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.635432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-systemd\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-var-lib-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.635520 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-run\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.635586 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:28.135562415 +0000 UTC m=+2.033237390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-run\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-systemd\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cnibin\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-var-lib-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635663 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cnibin\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jn55\" (UniqueName: \"kubernetes.io/projected/f137a282-0be2-47f7-aa1c-b2c5edae3249-kube-api-access-5jn55\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.636088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-sys\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.635855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfsd\" (UniqueName: \"kubernetes.io/projected/c08065a2-c657-476e-b036-7ce61e845062-kube-api-access-qhfsd\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-kubernetes\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-sys\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-lib-modules\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-etc-selinux\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-var-lib-kubelet\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-kubelet-config\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-netns\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-kubernetes\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-env-overrides\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-script-lib\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-cni-binary-copy\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.636772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-etc-kubernetes\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-var-lib-kubelet\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffe4393-063b-4937-97f4-35ace2c6f1d6-tmp-dir\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-run-netns\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-lib-modules\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.636869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c08065a2-c657-476e-b036-7ce61e845062-iptables-alerter-script\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-daemon-config\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-kubelet-config\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637191 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-etc-kubernetes\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysconfig\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-registration-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-systemd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637413 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-env-overrides\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vml\" (UniqueName: \"kubernetes.io/projected/96797814-df34-4f04-a4b0-8c054e5512d1-kube-api-access-82vml\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysconfig\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.637504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-registration-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637580 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-systemd\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-sys-fs\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-tuned\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-sys-fs\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fffe4393-063b-4937-97f4-35ace2c6f1d6-tmp-dir\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c08065a2-c657-476e-b036-7ce61e845062-iptables-alerter-script\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66x8m\" (UniqueName: \"kubernetes.io/projected/5dd4f240-48ff-4c88-b6c4-d219767ed76c-kube-api-access-66x8m\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-host\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.637945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cs5\" (UniqueName: \"kubernetes.io/projected/b39de062-6cbe-41fe-82e7-6133b46a5bca-kube-api-access-m2cs5\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-host\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-script-lib\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb496619-1f24-4fba-a0d2-cb1f942047df-agent-certs\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.638193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckm9\" (UniqueName: \"kubernetes.io/projected/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-kube-api-access-9ckm9\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-systemd-units\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvscw\" (UniqueName: \"kubernetes.io/projected/964bdece-796d-4db9-b435-0a59fa03921b-kube-api-access-gvscw\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-cnibin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.638418 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-socket-dir-parent\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-bin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.638571 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:28.138546617 +0000 UTC m=+2.036221606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-bin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638700 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-cnibin\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-systemd-units\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-socket-dir-parent\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-kubelet\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.638856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-bin\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-kubelet\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-cni-bin\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.638975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-multus\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-conf\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-var-lib-cni-multus\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-binary-copy\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639407 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-sysctl-conf\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-dbus\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-socket-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-os-release\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-os-release\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-cni-binary-copy\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-socket-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-conf-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04d71036-db1b-40aa-9924-d767d55533b4-dbus\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:27.639859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vl5\" (UniqueName: \"kubernetes.io/projected/144b6fd1-04ca-4c47-9093-ae50d3a088e0-kube-api-access-84vl5\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39de062-6cbe-41fe-82e7-6133b46a5bca-serviceca\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-ovn\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-log-socket\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.639966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96797814-df34-4f04-a4b0-8c054e5512d1-ovn-node-metrics-cert\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-log-socket\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-multus-certs\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-multus-certs\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffe4393-063b-4937-97f4-35ace2c6f1d6-hosts-file\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-conf-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-system-cni-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-etc-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-etc-openvswitch\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39de062-6cbe-41fe-82e7-6133b46a5bca-serviceca\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.640391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-config\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39de062-6cbe-41fe-82e7-6133b46a5bca-host\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-run-ovn\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-modprobe-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-device-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fffe4393-063b-4937-97f4-35ace2c6f1d6-hosts-file\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-kubelet\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-k8s-cni-cncf-io\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtht\" (UniqueName: \"kubernetes.io/projected/fffe4393-063b-4937-97f4-35ace2c6f1d6-kube-api-access-8dtht\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-os-release\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb496619-1f24-4fba-a0d2-cb1f942047df-konnectivity-ca\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-slash\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-slash\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39de062-6cbe-41fe-82e7-6133b46a5bca-host\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-tmp\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640862 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-modprobe-d\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-multus-cni-dir\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.641142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f137a282-0be2-47f7-aa1c-b2c5edae3249-host-run-k8s-cni-cncf-io\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-os-release\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.640968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/964bdece-796d-4db9-b435-0a59fa03921b-device-dir\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.641016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96797814-df34-4f04-a4b0-8c054e5512d1-host-kubelet\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.641063 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dd4f240-48ff-4c88-b6c4-d219767ed76c-system-cni-dir\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.641293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb496619-1f24-4fba-a0d2-cb1f942047df-agent-certs\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.641373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dd4f240-48ff-4c88-b6c4-d219767ed76c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.641963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.641468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb496619-1f24-4fba-a0d2-cb1f942047df-konnectivity-ca\") pod \"konnectivity-agent-xvmbh\" (UID: \"fb496619-1f24-4fba-a0d2-cb1f942047df\") " pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.642266 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.642213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96797814-df34-4f04-a4b0-8c054e5512d1-ovnkube-config\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.643089 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.642746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96797814-df34-4f04-a4b0-8c054e5512d1-ovn-node-metrics-cert\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.644065 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.643996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/144b6fd1-04ca-4c47-9093-ae50d3a088e0-etc-tuned\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.644493 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.644466 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:27.644593 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.644498 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:27.644593 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.644513 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:27.644593 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:27.644577 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:28.144560373 +0000 UTC m=+2.042235369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:27.646145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfsd\" (UniqueName: \"kubernetes.io/projected/c08065a2-c657-476e-b036-7ce61e845062-kube-api-access-qhfsd\") pod \"iptables-alerter-65bdl\" (UID: \"c08065a2-c657-476e-b036-7ce61e845062\") " pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.646318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cs5\" (UniqueName: \"kubernetes.io/projected/b39de062-6cbe-41fe-82e7-6133b46a5bca-kube-api-access-m2cs5\") pod \"node-ca-r7sls\" (UID: \"b39de062-6cbe-41fe-82e7-6133b46a5bca\") " pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.646430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vml\" (UniqueName: \"kubernetes.io/projected/96797814-df34-4f04-a4b0-8c054e5512d1-kube-api-access-82vml\") pod \"ovnkube-node-27mk5\" (UID: \"96797814-df34-4f04-a4b0-8c054e5512d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.646582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jn55\" (UniqueName: \"kubernetes.io/projected/f137a282-0be2-47f7-aa1c-b2c5edae3249-kube-api-access-5jn55\") pod \"multus-fls4b\" (UID: \"f137a282-0be2-47f7-aa1c-b2c5edae3249\") " pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.646746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvscw\" (UniqueName: \"kubernetes.io/projected/964bdece-796d-4db9-b435-0a59fa03921b-kube-api-access-gvscw\") pod \"aws-ebs-csi-driver-node-d8b56\" (UID: \"964bdece-796d-4db9-b435-0a59fa03921b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.646937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.646920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66x8m\" (UniqueName: \"kubernetes.io/projected/5dd4f240-48ff-4c88-b6c4-d219767ed76c-kube-api-access-66x8m\") pod \"multus-additional-cni-plugins-54ff4\" (UID: \"5dd4f240-48ff-4c88-b6c4-d219767ed76c\") " pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.647430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.647414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckm9\" (UniqueName: \"kubernetes.io/projected/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-kube-api-access-9ckm9\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:27.649628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.649609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vl5\" (UniqueName: \"kubernetes.io/projected/144b6fd1-04ca-4c47-9093-ae50d3a088e0-kube-api-access-84vl5\") pod \"tuned-rch2d\" (UID: \"144b6fd1-04ca-4c47-9093-ae50d3a088e0\") " pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.653186 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.653169 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtht\" (UniqueName: \"kubernetes.io/projected/fffe4393-063b-4937-97f4-35ace2c6f1d6-kube-api-access-8dtht\") pod \"node-resolver-lpdkp\" (UID: \"fffe4393-063b-4937-97f4-35ace2c6f1d6\") " pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.658905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.658887 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:27.756544 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.756471 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:27.762264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.762241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" Jul 26 15:05:27.765418 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.765387 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96797814_df34_4f04_a4b0_8c054e5512d1.slice/crio-5bf4768d5765cd99d396188149ca5f363b4bef144fb82ad70c4282594ec7a1cd WatchSource:0}: Error finding container 5bf4768d5765cd99d396188149ca5f363b4bef144fb82ad70c4282594ec7a1cd: Status 404 returned error can't find the container with id 5bf4768d5765cd99d396188149ca5f363b4bef144fb82ad70c4282594ec7a1cd Jul 26 15:05:27.770205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.770182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7sls" Jul 26 15:05:27.770980 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.770955 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964bdece_796d_4db9_b435_0a59fa03921b.slice/crio-b75abe91e896c05e0c8658a07a7f52672eb5862d66415599fa41e9542adbaa81 WatchSource:0}: Error finding container b75abe91e896c05e0c8658a07a7f52672eb5862d66415599fa41e9542adbaa81: Status 404 returned error can't find the container with id b75abe91e896c05e0c8658a07a7f52672eb5862d66415599fa41e9542adbaa81 Jul 26 15:05:27.773604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.773583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fls4b" Jul 26 15:05:27.779793 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.779769 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39de062_6cbe_41fe_82e7_6133b46a5bca.slice/crio-c317de93554c65ee3224687900cba8e64e8159a1f43f7b312e2bf33048230083 WatchSource:0}: Error finding container c317de93554c65ee3224687900cba8e64e8159a1f43f7b312e2bf33048230083: Status 404 returned error can't find the container with id c317de93554c65ee3224687900cba8e64e8159a1f43f7b312e2bf33048230083 Jul 26 15:05:27.780215 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.780184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-65bdl" Jul 26 15:05:27.782891 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.782861 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf137a282_0be2_47f7_aa1c_b2c5edae3249.slice/crio-a50499d5c945b0f009bb8a4981afbf175cd20c75897aad03fd3331ad86847f86 WatchSource:0}: Error finding container a50499d5c945b0f009bb8a4981afbf175cd20c75897aad03fd3331ad86847f86: Status 404 returned error can't find the container with id a50499d5c945b0f009bb8a4981afbf175cd20c75897aad03fd3331ad86847f86 Jul 26 15:05:27.786396 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.786379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:27.790908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.790890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rch2d" Jul 26 15:05:27.791325 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.791300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08065a2_c657_476e_b036_7ce61e845062.slice/crio-118f6aa908629b15f579c51ca895e8170d35c37dccf1c7d7a239d1a2226f7fca WatchSource:0}: Error finding container 118f6aa908629b15f579c51ca895e8170d35c37dccf1c7d7a239d1a2226f7fca: Status 404 returned error can't find the container with id 118f6aa908629b15f579c51ca895e8170d35c37dccf1c7d7a239d1a2226f7fca Jul 26 15:05:27.796346 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.796321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpdkp" Jul 26 15:05:27.796636 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.796609 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb496619_1f24_4fba_a0d2_cb1f942047df.slice/crio-05dab249cee04bbc23a89ebdaeb8c63c1a65d1e921f17e35e53db5b6022a3e49 WatchSource:0}: Error finding container 05dab249cee04bbc23a89ebdaeb8c63c1a65d1e921f17e35e53db5b6022a3e49: Status 404 returned error can't find the container with id 05dab249cee04bbc23a89ebdaeb8c63c1a65d1e921f17e35e53db5b6022a3e49 Jul 26 15:05:27.799808 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.799785 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144b6fd1_04ca_4c47_9093_ae50d3a088e0.slice/crio-3f4e80d2ee778370daed1ad385b1171d6827d26b39020d92a71f83723efed181 WatchSource:0}: Error finding container 3f4e80d2ee778370daed1ad385b1171d6827d26b39020d92a71f83723efed181: Status 404 returned error can't find the container with id 3f4e80d2ee778370daed1ad385b1171d6827d26b39020d92a71f83723efed181 Jul 26 15:05:27.800721 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:27.800700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54ff4" Jul 26 15:05:27.805738 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.805710 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffe4393_063b_4937_97f4_35ace2c6f1d6.slice/crio-4d8a831b3e5789dd0e1055200a04876619d2750223efe22c2b0cba61c324ed2a WatchSource:0}: Error finding container 4d8a831b3e5789dd0e1055200a04876619d2750223efe22c2b0cba61c324ed2a: Status 404 returned error can't find the container with id 4d8a831b3e5789dd0e1055200a04876619d2750223efe22c2b0cba61c324ed2a Jul 26 15:05:27.810828 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:05:27.810804 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd4f240_48ff_4c88_b6c4_d219767ed76c.slice/crio-4715994f612593f27a6fdf652266173472c1b956438954514a48f0b57127915b WatchSource:0}: Error finding container 4715994f612593f27a6fdf652266173472c1b956438954514a48f0b57127915b: Status 404 returned error can't find the container with id 4715994f612593f27a6fdf652266173472c1b956438954514a48f0b57127915b Jul 26 15:05:28.145855 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.145763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:28.145855 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.145820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.145871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.145921 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.145985 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.145965417 +0000 UTC m=+3.043640413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.145988 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.146018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.14601013 +0000 UTC m=+3.043685105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:28.146072 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.146065 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:28.146450 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.146082 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:28.146450 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.146094 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:28.146450 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:28.146145 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:29.146127853 +0000 UTC m=+3.043802871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:28.468183 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.468061 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:00:27 +0000 UTC" deadline="2028-04-19 22:51:37.078285348 +0000 UTC" Jul 26 15:05:28.468183 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.468126 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15199h46m8.610163226s" Jul 26 15:05:28.577293 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.577236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerStarted","Data":"4715994f612593f27a6fdf652266173472c1b956438954514a48f0b57127915b"} Jul 26 15:05:28.589301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.589203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvmbh" event={"ID":"fb496619-1f24-4fba-a0d2-cb1f942047df","Type":"ContainerStarted","Data":"05dab249cee04bbc23a89ebdaeb8c63c1a65d1e921f17e35e53db5b6022a3e49"} Jul 26 15:05:28.610287 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.610219 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65bdl" event={"ID":"c08065a2-c657-476e-b036-7ce61e845062","Type":"ContainerStarted","Data":"118f6aa908629b15f579c51ca895e8170d35c37dccf1c7d7a239d1a2226f7fca"} Jul 26 15:05:28.623239 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.623200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpdkp" event={"ID":"fffe4393-063b-4937-97f4-35ace2c6f1d6","Type":"ContainerStarted","Data":"4d8a831b3e5789dd0e1055200a04876619d2750223efe22c2b0cba61c324ed2a"} Jul 26 15:05:28.643159 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.643124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rch2d" event={"ID":"144b6fd1-04ca-4c47-9093-ae50d3a088e0","Type":"ContainerStarted","Data":"3f4e80d2ee778370daed1ad385b1171d6827d26b39020d92a71f83723efed181"} Jul 26 15:05:28.650318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.650261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fls4b" event={"ID":"f137a282-0be2-47f7-aa1c-b2c5edae3249","Type":"ContainerStarted","Data":"a50499d5c945b0f009bb8a4981afbf175cd20c75897aad03fd3331ad86847f86"} Jul 26 15:05:28.661016 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.660982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7sls" event={"ID":"b39de062-6cbe-41fe-82e7-6133b46a5bca","Type":"ContainerStarted","Data":"c317de93554c65ee3224687900cba8e64e8159a1f43f7b312e2bf33048230083"} Jul 26 15:05:28.671116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.670791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" event={"ID":"964bdece-796d-4db9-b435-0a59fa03921b","Type":"ContainerStarted","Data":"b75abe91e896c05e0c8658a07a7f52672eb5862d66415599fa41e9542adbaa81"} Jul 26 15:05:28.678073 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.678040 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"5bf4768d5765cd99d396188149ca5f363b4bef144fb82ad70c4282594ec7a1cd"} Jul 26 15:05:28.804448 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.804373 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:28.882294 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:28.881868 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.156053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.156165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.156204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156376 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156423 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156439 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156538 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:31.156518352 +0000 UTC m=+5.054193341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156938 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.156986 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:31.156971512 +0000 UTC m=+5.054646493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.157047 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:29.157185 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.157078 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:31.157068063 +0000 UTC m=+5.054743043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:29.469432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.469315 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:00:27 +0000 UTC" deadline="2028-03-10 04:33:34.823565455 +0000 UTC" Jul 26 15:05:29.469432 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.469383 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14221h28m5.354186597s" Jul 26 15:05:29.525962 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.525739 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.548559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.548686 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.549158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.549251 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:29.549335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:29.549477 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:29.549420 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.174504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.174571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.174615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174677 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174723 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174726 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174740 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174751 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:35.174737714 +0000 UTC m=+9.072412693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174781 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:35.174767716 +0000 UTC m=+9.072442695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:31.174802 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.174794 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:35.174788264 +0000 UTC m=+9.072463239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.548174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.548218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.548302 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:31.548314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.548409 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:31.548566 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:31.548482 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:33.548188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:33.548147 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:33.548662 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:33.548299 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:33.548662 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:33.548658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:33.548773 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:33.548751 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:33.548826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:33.548813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:33.549283 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:33.549249 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:35.209312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.209259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.209344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.209379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209402 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209465 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:43.209448584 +0000 UTC m=+17.107123559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209517 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209535 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209548 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209598 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:43.209581177 +0000 UTC m=+17.107256169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209662 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:35.209778 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.209697 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:43.209686174 +0000 UTC m=+17.107361168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.548977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.549120 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.548991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.549211 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:35.548981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:35.549679 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:35.549314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:37.548841 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:37.548744 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:37.548841 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:37.548772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:37.548841 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:37.548783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:37.549373 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:37.548896 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:37.549373 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:37.549023 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:37.549373 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:37.549088 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:39.548388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:39.548303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:39.548388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:39.548337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:39.548388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:39.548344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:39.548815 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:39.548411 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:39.548815 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:39.548486 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:39.548815 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:39.548576 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:41.548695 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:41.548661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:41.549152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:41.548664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:41.549152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:41.548774 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:41.549152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:41.548793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:41.549152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:41.548847 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:41.549152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:41.548973 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:43.265774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.265735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.265780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.265804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.265902 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.265968 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:59.265951866 +0000 UTC m=+33.163626845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.265902 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.266039 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:59.266027132 +0000 UTC m=+33.163702107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.265905 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.266058 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.266071 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:43.266152 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.266101 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:05:59.266094443 +0000 UTC m=+33.163769418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:43.548741 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.548652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:43.548905 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.548786 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:43.549186 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.549166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:43.549323 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.549294 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:43.549392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:43.549343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:43.549445 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:43.549390 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:45.548394 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:45.548348 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:45.548394 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:45.548386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:45.549008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:45.548385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:45.549008 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:45.548480 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:45.549008 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:45.548562 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:45.549008 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:45.548635 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:47.548909 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.548748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:47.549421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.548774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:47.549421 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:47.549012 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:47.549421 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:47.549079 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:47.549421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.548793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:47.549421 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:47.549200 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:47.732310 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.731045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" event={"ID":"a9f78f483deac2095511863fac8041de","Type":"ContainerStarted","Data":"496d239974345122adf99e6bc176f1595e187271a6db84cce8665c2aff56932f"} Jul 26 15:05:47.747020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.746574 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-15.ec2.internal" podStartSLOduration=20.746554409 podStartE2EDuration="20.746554409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:47.746321304 +0000 UTC m=+21.643996303" watchObservedRunningTime="2026-07-26 15:05:47.746554409 +0000 UTC m=+21.644229409" Jul 26 15:05:47.754078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.753668 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rch2d" event={"ID":"144b6fd1-04ca-4c47-9093-ae50d3a088e0","Type":"ContainerStarted","Data":"6d627e6fc5fb6d1e4d78e433f8cc5b94d6a7dc3cd017badfd9af0c510f83493b"} Jul 26 15:05:47.761782 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.761744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fls4b" event={"ID":"f137a282-0be2-47f7-aa1c-b2c5edae3249","Type":"ContainerStarted","Data":"06664d02281f5ab915a1ba08b8ecb8b183538873e568f6e7fa81eb1e0383d464"} Jul 26 15:05:47.769684 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.768383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"453a6f7e6953d53a7729664f151ca74b088aa985b974706d5aa4c4994657b784"} Jul 26 15:05:47.781380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.781338 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rch2d" podStartSLOduration=2.292229778 podStartE2EDuration="21.781321244s" totalImagesPullingTime="19.489091466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:47.780951495 +0000 UTC m=+21.678626493" watchObservedRunningTime="2026-07-26 15:05:47.781321244 +0000 UTC m=+21.678996242" Jul 26 15:05:47.814420 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:47.814325 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fls4b" podStartSLOduration=2.267624446 podStartE2EDuration="21.814300484s" totalImagesPullingTime="19.546676038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:47.813908869 +0000 UTC m=+21.711583887" watchObservedRunningTime="2026-07-26 15:05:47.814300484 +0000 UTC m=+21.711975483" Jul 26 15:05:48.771168 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.771121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7sls" event={"ID":"b39de062-6cbe-41fe-82e7-6133b46a5bca","Type":"ContainerStarted","Data":"324962bb7538ccf8d7a9a3e752023dc26717892ee109cbcee05085cb455a4998"} Jul 26 15:05:48.772526 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.772501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" event={"ID":"964bdece-796d-4db9-b435-0a59fa03921b","Type":"ContainerStarted","Data":"08520f98407b9c17c6d979a1ab3f49742d44df1db81ed592562765f839d63631"} Jul 26 15:05:48.775352 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.775327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"37e6521e79aff12ba31fcc7769c251b44bf5befe16780604d3db1a71a6ee5591"} Jul 26 15:05:48.775459 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.775404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"9a9203d98dfd6eaf219b285a21fc543e3c8985891576fe0bcfbcaeb0c1e39e1c"} Jul 26 15:05:48.775459 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.775447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"2c950c34afce90c81c654058cbc449b82a0c315f8805ffd41ffdad46d0533870"} Jul 26 15:05:48.775540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.775493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"7e68b9e80800fe956a846430c58fa2afc582e600850b372cbb54b99926fb87f1"} Jul 26 15:05:48.775540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.775535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"455ed6b5d16ae15e2c55091e794cf3ff52bd624bd0338c16d5a024de1b53381c"} Jul 26 15:05:48.776915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.776895 2575 generic.go:356] "Generic (PLEG): container finished" podID="0b6b5cf3c3dabcc7864a976f54f07afe" containerID="9ddd542d742c099f74f593a5e5ce0d056d744a9bc79e1ac26c8f7e3072b62804" exitCode=0 Jul 26 15:05:48.777005 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.776956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" event={"ID":"0b6b5cf3c3dabcc7864a976f54f07afe","Type":"ContainerDied","Data":"9ddd542d742c099f74f593a5e5ce0d056d744a9bc79e1ac26c8f7e3072b62804"} Jul 26 15:05:48.778563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.778540 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="1a781cd7102959ad072e7e820fc351d1c517732f31613c576e70d2630fd95b73" exitCode=0 Jul 26 15:05:48.778648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.778630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"1a781cd7102959ad072e7e820fc351d1c517732f31613c576e70d2630fd95b73"} Jul 26 15:05:48.780231 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.780214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvmbh" event={"ID":"fb496619-1f24-4fba-a0d2-cb1f942047df","Type":"ContainerStarted","Data":"c1fa69f5589d4b05e6cab19ae31daeaf8392708a2598b485a5e9f3af01fda311"} Jul 26 15:05:48.781556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.781535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpdkp" event={"ID":"fffe4393-063b-4937-97f4-35ace2c6f1d6","Type":"ContainerStarted","Data":"7336d316ba77ab9bb076e26a1bc19397975d273fd41c5ac735bb4d9dab1eeff0"} Jul 26 15:05:48.785496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.785461 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r7sls" podStartSLOduration=3.278273038 podStartE2EDuration="22.785448869s" totalImagesPullingTime="19.507175831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:48.785307382 +0000 UTC m=+22.682982376" watchObservedRunningTime="2026-07-26 15:05:48.785448869 +0000 UTC m=+22.683123867" Jul 26 15:05:48.814069 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.813839 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xvmbh" podStartSLOduration=7.5872932429999995 podStartE2EDuration="22.813822223s" totalImagesPullingTime="15.22652898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:48.813598276 +0000 UTC m=+22.711273278" watchObservedRunningTime="2026-07-26 15:05:48.813822223 +0000 UTC m=+22.711497221" Jul 26 15:05:48.826902 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:48.826855 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lpdkp" podStartSLOduration=3.343436627 podStartE2EDuration="22.826842086s" totalImagesPullingTime="19.483405459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:48.826801521 +0000 UTC m=+22.724476517" watchObservedRunningTime="2026-07-26 15:05:48.826842086 +0000 UTC m=+22.724517082" Jul 26 15:05:49.184611 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.184584 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 15:05:49.498354 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.498192 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T15:05:49.184601453Z","UUID":"1ee34124-b643-4ee2-9325-a9a073134294","Handler":null,"Name":"","Endpoint":""} Jul 26 15:05:49.500154 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.500136 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 15:05:49.500317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.500177 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 15:05:49.548026 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.547979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:49.548203 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:49.548109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:49.548540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.548521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:49.548652 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:49.548634 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:49.548736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.548724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:49.548838 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:49.548816 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:49.790804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.790717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-65bdl" event={"ID":"c08065a2-c657-476e-b036-7ce61e845062","Type":"ContainerStarted","Data":"8a83efae74e518a2d46196c78f16e49c6498016c9d4b04bb16f55dae814ca340"} Jul 26 15:05:49.792965 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.792935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" event={"ID":"964bdece-796d-4db9-b435-0a59fa03921b","Type":"ContainerStarted","Data":"a2de17b85193faf1bdd2507ac68dd08c5393dab1d46942e75cf3f1828f62f363"} Jul 26 15:05:49.795445 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.795379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" event={"ID":"0b6b5cf3c3dabcc7864a976f54f07afe","Type":"ContainerStarted","Data":"1b30eeee1add86a0ff86372e16b6715dc3920c764b841831fb9ff259b7d12fd0"} Jul 26 15:05:49.804897 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.804840 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-65bdl" podStartSLOduration=4.329605831 podStartE2EDuration="23.804822036s" totalImagesPullingTime="19.475216205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:49.80413744 +0000 UTC m=+23.701812439" watchObservedRunningTime="2026-07-26 15:05:49.804822036 +0000 UTC m=+23.702497037" Jul 26 15:05:49.818369 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:49.818322 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-15.ec2.internal" podStartSLOduration=22.8183054 podStartE2EDuration="22.8183054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:49.81758683 +0000 UTC m=+23.715261829" watchObservedRunningTime="2026-07-26 15:05:49.8183054 +0000 UTC m=+23.715980434" Jul 26 15:05:50.800795 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:50.800754 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" event={"ID":"964bdece-796d-4db9-b435-0a59fa03921b","Type":"ContainerStarted","Data":"5c36f0ce0568618d7478ca2bc8b14d80f7d4822d92008f5e09b2076d2788e963"} Jul 26 15:05:50.804466 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:50.804435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"04db208b97ff5c15e7b53a4f9a6675952b3afa7b8a14eac95a00d64d1bb2763b"} Jul 26 15:05:51.548704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:51.548662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:51.548912 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:51.548670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:51.548912 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:51.548782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:51.548912 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:51.548844 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:51.548912 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:51.548670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:51.549088 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:51.548971 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:52.422519 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:52.422482 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:52.423313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:52.423289 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:52.437517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:52.437465 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d8b56" podStartSLOduration=4.883680923 podStartE2EDuration="26.43745011s" totalImagesPullingTime="21.553769187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:50.819919814 +0000 UTC m=+24.717594810" watchObservedRunningTime="2026-07-26 15:05:52.43745011 +0000 UTC m=+26.335125146" Jul 26 15:05:52.811638 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:52.811519 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:52.811990 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:52.811972 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvmbh" Jul 26 15:05:53.548812 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.548635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:53.549476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.548657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:53.549476 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:53.548880 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:53.549476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.548705 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:53.549476 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:53.548979 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:53.549476 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:53.549058 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:53.813576 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.813477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" event={"ID":"96797814-df34-4f04-a4b0-8c054e5512d1","Type":"ContainerStarted","Data":"cff04b0eeb98e0b6832e4a82ca571c2a6be23c347aa37eed6106f9cb6524542a"} Jul 26 15:05:53.813777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.813760 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:53.813870 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.813833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:53.815682 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.815646 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="b0883ccf3000d2f454cff5b8fb3bd9df8057a0660148e88c691a2b3e3294af66" exitCode=0 Jul 26 15:05:53.815827 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.815724 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"b0883ccf3000d2f454cff5b8fb3bd9df8057a0660148e88c691a2b3e3294af66"} Jul 26 15:05:53.830079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.830053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:53.830333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.830317 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:53.842412 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:53.842370 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" podStartSLOduration=7.993575128 podStartE2EDuration="27.84235462s" totalImagesPullingTime="19.848779492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:05:53.840839345 +0000 UTC m=+27.738514353" watchObservedRunningTime="2026-07-26 15:05:53.84235462 +0000 UTC m=+27.740029617" Jul 26 15:05:54.818243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:54.818215 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 26 15:05:55.080978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.080674 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k44rb"] Jul 26 15:05:55.080978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.080850 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:55.081192 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:55.080969 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:55.081702 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.081681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j6cxn"] Jul 26 15:05:55.081822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.081781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:55.081883 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:55.081867 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:55.082558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.082536 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ts4r"] Jul 26 15:05:55.082663 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.082642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:55.082775 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:55.082753 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:55.620336 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.620301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:05:55.822844 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.822811 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="bfb9c436875157d9a9733cfd6b8d4a6a8e8a785cd0f1c0aa6d675ab39b983741" exitCode=0 Jul 26 15:05:55.823205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:55.822893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"bfb9c436875157d9a9733cfd6b8d4a6a8e8a785cd0f1c0aa6d675ab39b983741"} Jul 26 15:05:56.549878 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:56.549843 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:56.550054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:56.549924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:56.550054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:56.549956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:56.550054 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:56.549978 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:56.550054 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:56.550045 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:56.550195 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:56.550133 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:56.828427 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:56.828381 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="fbcbbf22bee5d105035cb376708e7c6ed601402443826b7a0f5e322f85002cc8" exitCode=0 Jul 26 15:05:56.828871 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:56.828463 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"fbcbbf22bee5d105035cb376708e7c6ed601402443826b7a0f5e322f85002cc8"} Jul 26 15:05:58.549053 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:58.549015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:58.549550 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:58.549158 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k44rb" podUID="04d71036-db1b-40aa-9924-d767d55533b4" Jul 26 15:05:58.549616 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:58.549573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:58.549884 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:58.549677 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:05:58.549884 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:58.549778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:58.549884 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:58.549853 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j6cxn" podUID="5d5756a0-be23-4242-976b-270e869bc698" Jul 26 15:05:59.286664 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:59.286617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:59.286704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:05:59.286749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286823 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286841 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286852 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286863 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286902 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret podName:04d71036-db1b-40aa-9924-d767d55533b4 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:31.286884871 +0000 UTC m=+65.184559848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret") pod "global-pull-secret-syncer-k44rb" (UID: "04d71036-db1b-40aa-9924-d767d55533b4") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286866 2575 projected.go:196] Error preparing data for projected volume kube-api-access-f2bqf for pod openshift-network-diagnostics/network-check-target-j6cxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286918 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:31.286910671 +0000 UTC m=+65.184585645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:05:59.286934 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:05:59.286941 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf podName:5d5756a0-be23-4242-976b-270e869bc698 nodeName:}" failed. No retries permitted until 2026-07-26 15:06:31.28692888 +0000 UTC m=+65.184603857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2bqf" (UniqueName: "kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf") pod "network-check-target-j6cxn" (UID: "5d5756a0-be23-4242-976b-270e869bc698") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:06:00.403830 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.403761 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-15.ec2.internal" event="NodeReady" Jul 26 15:06:00.404327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.403891 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 15:06:00.446018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.445974 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4s5l"] Jul 26 15:06:00.476670 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.476635 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j967z"] Jul 26 15:06:00.476835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.476772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.479352 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.479321 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 15:06:00.479496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.479358 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 15:06:00.479496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.479392 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8qc59\"" Jul 26 15:06:00.498158 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.498131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4s5l"] Jul 26 15:06:00.498311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.498176 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j967z"] Jul 26 15:06:00.498311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.498293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.500885 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.500861 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 15:06:00.501023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.500894 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 15:06:00.501023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.500914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fz4kh\"" Jul 26 15:06:00.501023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.500921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 15:06:00.548329 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.548295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:06:00.548329 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.548321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:06:00.548771 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.548756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:06:00.550879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.550843 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:06:00.551010 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.550930 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:06:00.551081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.551062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgnb2\"" Jul 26 15:06:00.551138 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.551085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4dzfg\"" Jul 26 15:06:00.551190 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.551170 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:06:00.551409 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.551391 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:06:00.595863 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.595822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96663026-7fcd-4267-a3f8-988a24bacacc-tmp-dir\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.596041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.595880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eab08770-3c75-4fad-bdeb-c5615fca0def-cert\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.596041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.595934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.596041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.596003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96663026-7fcd-4267-a3f8-988a24bacacc-config-volume\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.596206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.596042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jbf\" (UniqueName: \"kubernetes.io/projected/96663026-7fcd-4267-a3f8-988a24bacacc-kube-api-access-g9jbf\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.596206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.596063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jkg\" (UniqueName: \"kubernetes.io/projected/eab08770-3c75-4fad-bdeb-c5615fca0def-kube-api-access-q9jkg\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.697159 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jbf\" (UniqueName: \"kubernetes.io/projected/96663026-7fcd-4267-a3f8-988a24bacacc-kube-api-access-g9jbf\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.697159 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jkg\" (UniqueName: \"kubernetes.io/projected/eab08770-3c75-4fad-bdeb-c5615fca0def-kube-api-access-q9jkg\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.697426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96663026-7fcd-4267-a3f8-988a24bacacc-tmp-dir\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.697426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eab08770-3c75-4fad-bdeb-c5615fca0def-cert\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.697426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.697426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96663026-7fcd-4267-a3f8-988a24bacacc-config-volume\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.697655 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:00.697429 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:00.697655 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:00.697513 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:01.197491079 +0000 UTC m=+35.095166084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:00.697726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.697661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96663026-7fcd-4267-a3f8-988a24bacacc-tmp-dir\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.698027 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.698008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96663026-7fcd-4267-a3f8-988a24bacacc-config-volume\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.701525 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.701418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eab08770-3c75-4fad-bdeb-c5615fca0def-cert\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.705863 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.705839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jbf\" (UniqueName: \"kubernetes.io/projected/96663026-7fcd-4267-a3f8-988a24bacacc-kube-api-access-g9jbf\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:00.705976 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.705925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jkg\" (UniqueName: \"kubernetes.io/projected/eab08770-3c75-4fad-bdeb-c5615fca0def-kube-api-access-q9jkg\") pod \"ingress-canary-j967z\" (UID: \"eab08770-3c75-4fad-bdeb-c5615fca0def\") " pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:00.808670 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:00.808636 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j967z" Jul 26 15:06:01.180784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:01.180751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j967z"] Jul 26 15:06:01.184474 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:06:01.184440 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab08770_3c75_4fad_bdeb_c5615fca0def.slice/crio-18185df31ebbe0fa7183b4294b7f6185bae31b580752634d48ef36587b1de5e3 WatchSource:0}: Error finding container 18185df31ebbe0fa7183b4294b7f6185bae31b580752634d48ef36587b1de5e3: Status 404 returned error can't find the container with id 18185df31ebbe0fa7183b4294b7f6185bae31b580752634d48ef36587b1de5e3 Jul 26 15:06:01.201703 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:01.201665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:01.201840 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:01.201818 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:01.201902 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:01.201883 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:02.201865914 +0000 UTC m=+36.099540889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:01.843454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:01.843414 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="b9c1ad1bfac63ecbe7a8cfe7db7b79c73865ff4190ad6871745bdb203f1258cc" exitCode=0 Jul 26 15:06:01.844181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:01.843499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"b9c1ad1bfac63ecbe7a8cfe7db7b79c73865ff4190ad6871745bdb203f1258cc"} Jul 26 15:06:01.845022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:01.844994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j967z" event={"ID":"eab08770-3c75-4fad-bdeb-c5615fca0def","Type":"ContainerStarted","Data":"18185df31ebbe0fa7183b4294b7f6185bae31b580752634d48ef36587b1de5e3"} Jul 26 15:06:02.209311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:02.209267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:02.209794 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:02.209761 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:02.209927 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:02.209876 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:04.20985248 +0000 UTC m=+38.107527468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:02.850744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:02.850712 2575 generic.go:356] "Generic (PLEG): container finished" podID="5dd4f240-48ff-4c88-b6c4-d219767ed76c" containerID="6eb9687556eeebca0f7c5d63e9be56b6b01d8f84271939205f24f4ad714ee3f4" exitCode=0 Jul 26 15:06:02.851213 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:02.850769 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerDied","Data":"6eb9687556eeebca0f7c5d63e9be56b6b01d8f84271939205f24f4ad714ee3f4"} Jul 26 15:06:03.857197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:03.856991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54ff4" event={"ID":"5dd4f240-48ff-4c88-b6c4-d219767ed76c","Type":"ContainerStarted","Data":"6c0051659ed15cf8d50651dfa8e973d4146db0748586ed2d72bd419f2d836641"} Jul 26 15:06:03.858465 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:03.858431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j967z" event={"ID":"eab08770-3c75-4fad-bdeb-c5615fca0def","Type":"ContainerStarted","Data":"f8b00590be2234f1ed70b24656b63307129edf3561a25273bd1f20a86bb12f8c"} Jul 26 15:06:03.881294 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:03.881226 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-54ff4" podStartSLOduration=8.350635645 podStartE2EDuration="37.881210591s" totalImagesPullingTime="29.530574946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:03.880702552 +0000 UTC m=+37.778377558" watchObservedRunningTime="2026-07-26 15:06:03.881210591 +0000 UTC m=+37.778885587" Jul 26 15:06:03.894954 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:03.894896 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j967z" podStartSLOduration=1.416905096 podStartE2EDuration="3.894879218s" totalImagesPullingTime="2.477974122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:03.894217392 +0000 UTC m=+37.791892390" watchObservedRunningTime="2026-07-26 15:06:03.894879218 +0000 UTC m=+37.792554215" Jul 26 15:06:04.228356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:04.228310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:04.228538 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:04.228439 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:04.228538 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:04.228515 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:08.228497678 +0000 UTC m=+42.126172669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:08.253877 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:08.253833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:08.254423 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:08.254007 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:08.254423 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:08.254100 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:16.254077516 +0000 UTC m=+50.151752492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:16.306455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:16.306411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:16.306916 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:16.306569 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:16.306916 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:16.306630 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:06:32.306614742 +0000 UTC m=+66.204289717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:26.840143 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:26.840115 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27mk5" Jul 26 15:06:31.309205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.309163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:06:31.309205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.309213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:06:31.309726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.309248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:06:31.312082 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.312063 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:06:31.312289 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.312260 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:06:31.312340 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.312305 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:06:31.319622 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:31.319604 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:06:31.319721 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:31.319669 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:07:35.319652098 +0000 UTC m=+129.217327087 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : secret "metrics-daemon-secret" not found Jul 26 15:06:31.321787 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.321769 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:06:31.323043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.323023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04d71036-db1b-40aa-9924-d767d55533b4-original-pull-secret\") pod \"global-pull-secret-syncer-k44rb\" (UID: \"04d71036-db1b-40aa-9924-d767d55533b4\") " pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:06:31.332863 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.332840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bqf\" (UniqueName: \"kubernetes.io/projected/5d5756a0-be23-4242-976b-270e869bc698-kube-api-access-f2bqf\") pod \"network-check-target-j6cxn\" (UID: \"5d5756a0-be23-4242-976b-270e869bc698\") " pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:06:31.462671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.462637 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4dzfg\"" Jul 26 15:06:31.470002 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.469971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:06:31.472311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.472266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k44rb" Jul 26 15:06:31.625383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.625352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j6cxn"] Jul 26 15:06:31.628759 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:06:31.628723 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5756a0_be23_4242_976b_270e869bc698.slice/crio-61b226616964fdee407a0ac99ebfdfe3d8caa1372c825e5eee849d06b3b97ddb WatchSource:0}: Error finding container 61b226616964fdee407a0ac99ebfdfe3d8caa1372c825e5eee849d06b3b97ddb: Status 404 returned error can't find the container with id 61b226616964fdee407a0ac99ebfdfe3d8caa1372c825e5eee849d06b3b97ddb Jul 26 15:06:31.643879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.643813 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k44rb"] Jul 26 15:06:31.648734 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:06:31.648703 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d71036_db1b_40aa_9924_d767d55533b4.slice/crio-4e2d24a60923987dbdda788a4a31503624d88fa4af490cb38c4c2b55f08cbe23 WatchSource:0}: Error finding container 4e2d24a60923987dbdda788a4a31503624d88fa4af490cb38c4c2b55f08cbe23: Status 404 returned error can't find the container with id 4e2d24a60923987dbdda788a4a31503624d88fa4af490cb38c4c2b55f08cbe23 Jul 26 15:06:31.929023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.928988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k44rb" event={"ID":"04d71036-db1b-40aa-9924-d767d55533b4","Type":"ContainerStarted","Data":"4e2d24a60923987dbdda788a4a31503624d88fa4af490cb38c4c2b55f08cbe23"} Jul 26 15:06:31.929957 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:31.929935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j6cxn" event={"ID":"5d5756a0-be23-4242-976b-270e869bc698","Type":"ContainerStarted","Data":"61b226616964fdee407a0ac99ebfdfe3d8caa1372c825e5eee849d06b3b97ddb"} Jul 26 15:06:32.316427 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:32.316342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:06:32.316866 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:32.316494 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:06:32.316866 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:06:32.316559 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:07:04.316536965 +0000 UTC m=+98.214211940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:06:36.944010 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:36.943970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k44rb" event={"ID":"04d71036-db1b-40aa-9924-d767d55533b4","Type":"ContainerStarted","Data":"d588f09ef84265a6fb46f4f3bfaabd8ad648e8213f8ff67dde2971cbcaf8b0e4"} Jul 26 15:06:36.945365 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:36.945342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j6cxn" event={"ID":"5d5756a0-be23-4242-976b-270e869bc698","Type":"ContainerStarted","Data":"c32a48e6d30245a05b30cf9ea2be934ee3dce69ee9a3e5e15211207fff576249"} Jul 26 15:06:36.945467 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:36.945455 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:06:36.958914 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:36.958860 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k44rb" podStartSLOduration=66.655588329 podStartE2EDuration="1m10.958841012s" totalImagesPullingTime="4.303252683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:36.958765322 +0000 UTC m=+70.856440318" watchObservedRunningTime="2026-07-26 15:06:36.958841012 +0000 UTC m=+70.856516010" Jul 26 15:06:36.973108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:36.973064 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j6cxn" podStartSLOduration=66.656886723 podStartE2EDuration="1m10.973039671s" totalImagesPullingTime="4.316152948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:36.972461973 +0000 UTC m=+70.870136967" watchObservedRunningTime="2026-07-26 15:06:36.973039671 +0000 UTC m=+70.870714667" Jul 26 15:06:45.439982 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.439944 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q"] Jul 26 15:06:45.442832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.442813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.445453 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.445430 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 26 15:06:45.446107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 26 15:06:45.446239 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446221 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 26 15:06:45.446318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446298 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 26 15:06:45.446554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 26 15:06:45.446889 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 26 15:06:45.446966 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.446881 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 26 15:06:45.452813 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.452794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q"] Jul 26 15:06:45.503947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.503911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.503947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.503952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.504206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.503972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.504206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.503994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a1cfa1-33ce-48a3-87b5-f10f88373f78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.504206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.504036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzxqx\" (UniqueName: \"kubernetes.io/projected/88a1cfa1-33ce-48a3-87b5-f10f88373f78-kube-api-access-qzxqx\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.504206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.504113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604760 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604760 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a1cfa1-33ce-48a3-87b5-f10f88373f78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzxqx\" (UniqueName: \"kubernetes.io/projected/88a1cfa1-33ce-48a3-87b5-f10f88373f78-kube-api-access-qzxqx\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.604996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.604888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.605967 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.605933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a1cfa1-33ce-48a3-87b5-f10f88373f78-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.607436 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.607406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-ca\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.607594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.607573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.607747 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.607727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.607803 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.607736 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88a1cfa1-33ce-48a3-87b5-f10f88373f78-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.613062 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.613039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzxqx\" (UniqueName: \"kubernetes.io/projected/88a1cfa1-33ce-48a3-87b5-f10f88373f78-kube-api-access-qzxqx\") pod \"cluster-proxy-proxy-agent-7f86c557f7-c9h2q\" (UID: \"88a1cfa1-33ce-48a3-87b5-f10f88373f78\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.779347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.779246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:06:45.907443 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.907413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q"] Jul 26 15:06:45.910774 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:06:45.910732 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88a1cfa1_33ce_48a3_87b5_f10f88373f78.slice/crio-3901e086abc6946aeda454349b5133613e1ee72efcf3db0342024379522e2281 WatchSource:0}: Error finding container 3901e086abc6946aeda454349b5133613e1ee72efcf3db0342024379522e2281: Status 404 returned error can't find the container with id 3901e086abc6946aeda454349b5133613e1ee72efcf3db0342024379522e2281 Jul 26 15:06:45.967622 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:45.967581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerStarted","Data":"3901e086abc6946aeda454349b5133613e1ee72efcf3db0342024379522e2281"} Jul 26 15:06:49.984107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:49.984069 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerStarted","Data":"d1ada27ce8bbcd15912b4f8237137ced307d99cc91ba5af2538ee3df9f0d2a70"} Jul 26 15:06:49.984582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:49.984118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerStarted","Data":"b20f2f1d6b07a3eeb06a80c20ee3058ec71d261cd6e8356a4a222436214e2a94"} Jul 26 15:06:49.984582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:49.984138 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerStarted","Data":"1ec4656d0ad9b299d17a33f610a613211ca8364a0197fb17e8fba09ba847f9db"} Jul 26 15:06:50.002256 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:06:50.002205 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" podStartSLOduration=1.249956183 podStartE2EDuration="5.002191115s" totalImagesPullingTime="3.752234932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:06:50.000461727 +0000 UTC m=+83.898136724" watchObservedRunningTime="2026-07-26 15:06:50.002191115 +0000 UTC m=+83.899866112" Jul 26 15:07:04.334044 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:07:04.333999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:07:04.334536 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:07:04.334150 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:07:04.334536 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:07:04.334215 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:08:08.334198519 +0000 UTC m=+162.231873494 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:07:07.950666 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:07:07.950629 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j6cxn" Jul 26 15:07:35.353387 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:07:35.353346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:07:35.353864 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:07:35.353488 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:07:35.353864 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:07:35.353550 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:37.353534792 +0000 UTC m=+251.251209767 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : secret "metrics-daemon-secret" not found Jul 26 15:08:03.488241 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:08:03.488192 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z4s5l" podUID="96663026-7fcd-4267-a3f8-988a24bacacc" Jul 26 15:08:03.565837 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:08:03.565777 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:08:04.197411 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:08:04.197382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:08:08.384802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:08:08.384762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:08:08.385196 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:08:08.384883 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:08:08.385196 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:08:08.384945 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls podName:96663026-7fcd-4267-a3f8-988a24bacacc nodeName:}" failed. No retries permitted until 2026-07-26 15:10:10.38493061 +0000 UTC m=+284.282605586 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls") pod "dns-default-z4s5l" (UID: "96663026-7fcd-4267-a3f8-988a24bacacc") : secret "dns-default-metrics-tls" not found Jul 26 15:08:16.549435 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:08:16.549399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:09:36.535611 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.535578 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-n9tjl"] Jul 26 15:09:36.538491 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.538471 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.541166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.541142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 26 15:09:36.541258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.541180 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 26 15:09:36.542658 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.542638 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j7gzb\"" Jul 26 15:09:36.542744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.542724 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 26 15:09:36.544024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.544004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:09:36.547743 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.547723 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 26 15:09:36.551210 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.551191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-n9tjl"] Jul 26 15:09:36.688520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.688475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715bac81-2670-4893-ba37-c3388bacf589-serving-cert\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.688520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.688522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9l9\" (UniqueName: \"kubernetes.io/projected/715bac81-2670-4893-ba37-c3388bacf589-kube-api-access-5c9l9\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.688727 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.688539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-trusted-ca\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.688727 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.688689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-config\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.789303 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.789197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-config\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.789303 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.789266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715bac81-2670-4893-ba37-c3388bacf589-serving-cert\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.789513 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.789319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9l9\" (UniqueName: \"kubernetes.io/projected/715bac81-2670-4893-ba37-c3388bacf589-kube-api-access-5c9l9\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.789513 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.789344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-trusted-ca\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.789845 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.789819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-config\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.790134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.790112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/715bac81-2670-4893-ba37-c3388bacf589-trusted-ca\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.791657 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.791639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/715bac81-2670-4893-ba37-c3388bacf589-serving-cert\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.798100 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.798080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9l9\" (UniqueName: \"kubernetes.io/projected/715bac81-2670-4893-ba37-c3388bacf589-kube-api-access-5c9l9\") pod \"console-operator-584dcb6b54-n9tjl\" (UID: \"715bac81-2670-4893-ba37-c3388bacf589\") " pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.848176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.848126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:36.973142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:36.973110 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-n9tjl"] Jul 26 15:09:36.976573 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:09:36.976543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715bac81_2670_4893_ba37_c3388bacf589.slice/crio-e6b28094041a2d28a1c20454c91433de0d4d51bc534de336a551f7aecc782493 WatchSource:0}: Error finding container e6b28094041a2d28a1c20454c91433de0d4d51bc534de336a551f7aecc782493: Status 404 returned error can't find the container with id e6b28094041a2d28a1c20454c91433de0d4d51bc534de336a551f7aecc782493 Jul 26 15:09:37.393380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:37.393343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:09:37.393558 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:37.393483 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 26 15:09:37.393558 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:37.393534 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs podName:89b1a676-2fb9-4294-8a5a-6fd4e0c83da2 nodeName:}" failed. No retries permitted until 2026-07-26 15:11:39.39352075 +0000 UTC m=+373.291195725 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs") pod "network-metrics-daemon-5ts4r" (UID: "89b1a676-2fb9-4294-8a5a-6fd4e0c83da2") : secret "metrics-daemon-secret" not found Jul 26 15:09:37.454903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:37.454866 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" event={"ID":"715bac81-2670-4893-ba37-c3388bacf589","Type":"ContainerStarted","Data":"e6b28094041a2d28a1c20454c91433de0d4d51bc534de336a551f7aecc782493"} Jul 26 15:09:39.461987 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:39.461960 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/0.log" Jul 26 15:09:39.462484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:39.462000 2575 generic.go:356] "Generic (PLEG): container finished" podID="715bac81-2670-4893-ba37-c3388bacf589" containerID="3b464d019e2a154a3ae17f4470c93ffa75dda38dbf02b6d0dc65304f02ca0234" exitCode=255 Jul 26 15:09:39.462484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:39.462026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" event={"ID":"715bac81-2670-4893-ba37-c3388bacf589","Type":"ContainerDied","Data":"3b464d019e2a154a3ae17f4470c93ffa75dda38dbf02b6d0dc65304f02ca0234"} Jul 26 15:09:39.462484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:39.462348 2575 scope.go:117] "RemoveContainer" containerID="3b464d019e2a154a3ae17f4470c93ffa75dda38dbf02b6d0dc65304f02ca0234" Jul 26 15:09:40.468007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.467976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:09:40.468413 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.468392 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/0.log" Jul 26 15:09:40.468458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.468428 2575 generic.go:356] "Generic (PLEG): container finished" podID="715bac81-2670-4893-ba37-c3388bacf589" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" exitCode=255 Jul 26 15:09:40.468492 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.468472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" event={"ID":"715bac81-2670-4893-ba37-c3388bacf589","Type":"ContainerDied","Data":"30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87"} Jul 26 15:09:40.468537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.468525 2575 scope.go:117] "RemoveContainer" containerID="3b464d019e2a154a3ae17f4470c93ffa75dda38dbf02b6d0dc65304f02ca0234" Jul 26 15:09:40.468747 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:40.468723 2575 scope.go:117] "RemoveContainer" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" Jul 26 15:09:40.468924 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:40.468905 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-n9tjl_openshift-console-operator(715bac81-2670-4893-ba37-c3388bacf589)\"" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" podUID="715bac81-2670-4893-ba37-c3388bacf589" Jul 26 15:09:41.006846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.006805 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jwj6g"] Jul 26 15:09:41.011228 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.011207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.013959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.013934 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 15:09:41.014114 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.014095 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4q6m\"" Jul 26 15:09:41.014114 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.014105 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 15:09:41.014307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.014178 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 15:09:41.014427 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.014412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 15:09:41.027586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.027561 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwj6g"] Jul 26 15:09:41.117605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.117565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62666105-e03c-446e-87dc-06c069722fa9-crio-socket\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.117605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.117602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.117830 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.117694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.117830 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.117726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k87c\" (UniqueName: \"kubernetes.io/projected/62666105-e03c-446e-87dc-06c069722fa9-kube-api-access-4k87c\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.117830 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.117746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62666105-e03c-446e-87dc-06c069722fa9-data-volume\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.218696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62666105-e03c-446e-87dc-06c069722fa9-data-volume\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.218865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62666105-e03c-446e-87dc-06c069722fa9-crio-socket\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.218865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.218865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.218865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k87c\" (UniqueName: \"kubernetes.io/projected/62666105-e03c-446e-87dc-06c069722fa9-kube-api-access-4k87c\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62666105-e03c-446e-87dc-06c069722fa9-crio-socket\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.218901 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.218936 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.218966 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:41.718950102 +0000 UTC m=+255.616625076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.218970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62666105-e03c-446e-87dc-06c069722fa9-data-volume\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.219034 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.219016 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:41.718974811 +0000 UTC m=+255.616649810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : configmap "kube-rbac-proxy" not found Jul 26 15:09:41.229975 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.229950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k87c\" (UniqueName: \"kubernetes.io/projected/62666105-e03c-446e-87dc-06c069722fa9-kube-api-access-4k87c\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.472415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.472384 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:09:41.472784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.472702 2575 scope.go:117] "RemoveContainer" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" Jul 26 15:09:41.472861 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.472842 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-n9tjl_openshift-console-operator(715bac81-2670-4893-ba37-c3388bacf589)\"" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" podUID="715bac81-2670-4893-ba37-c3388bacf589" Jul 26 15:09:41.723032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.722931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.723032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:41.723031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:41.723320 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.723094 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.723320 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.723145 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:41.723320 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.723161 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:42.723143566 +0000 UTC m=+256.620818541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:41.723320 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:41.723198 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:42.723184109 +0000 UTC m=+256.620859084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : configmap "kube-rbac-proxy" not found Jul 26 15:09:42.730744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:42.730703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:42.731159 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:42.730773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:42.731159 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:42.730813 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:42.731159 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:42.730864 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:42.731159 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:42.730875 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:44.730861477 +0000 UTC m=+258.628536452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : configmap "kube-rbac-proxy" not found Jul 26 15:09:42.731159 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:42.730903 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:44.730892269 +0000 UTC m=+258.628567243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:42.806235 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:42.806204 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpdkp_fffe4393-063b-4937-97f4-35ace2c6f1d6/dns-node-resolver/0.log" Jul 26 15:09:44.208796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:44.208768 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7sls_b39de062-6cbe-41fe-82e7-6133b46a5bca/node-ca/0.log" Jul 26 15:09:44.747524 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:44.747488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:44.747710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:44.747549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:44.747710 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:44.747631 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:44.747822 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:44.747711 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:48.74769048 +0000 UTC m=+262.645365468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : configmap "kube-rbac-proxy" not found Jul 26 15:09:44.747822 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:44.747636 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:44.747822 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:44.747766 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:48.747754872 +0000 UTC m=+262.645429846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:44.806388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:44.806355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j967z_eab08770-3c75-4fad-bdeb-c5615fca0def/serve-healthcheck-canary/0.log" Jul 26 15:09:46.848650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:46.848522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:46.848650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:46.848624 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:09:46.849023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:46.848894 2575 scope.go:117] "RemoveContainer" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" Jul 26 15:09:46.849066 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:46.849030 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-n9tjl_openshift-console-operator(715bac81-2670-4893-ba37-c3388bacf589)\"" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" podUID="715bac81-2670-4893-ba37-c3388bacf589" Jul 26 15:09:47.489748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:47.489721 2575 scope.go:117] "RemoveContainer" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" Jul 26 15:09:47.489905 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:47.489870 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-n9tjl_openshift-console-operator(715bac81-2670-4893-ba37-c3388bacf589)\"" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" podUID="715bac81-2670-4893-ba37-c3388bacf589" Jul 26 15:09:48.776240 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:48.776193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:48.776639 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:48.776297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:48.776639 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:48.776374 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:48.776639 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:48.776408 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:09:48.776639 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:48.776440 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:56.776424073 +0000 UTC m=+270.674099073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:48.776639 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:48.776463 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:09:56.776448976 +0000 UTC m=+270.674123951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : configmap "kube-rbac-proxy" not found Jul 26 15:09:56.837175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:56.837132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:56.837647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:56.837208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:09:56.837647 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:56.837309 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:09:56.837647 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:09:56.837376 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls podName:62666105-e03c-446e-87dc-06c069722fa9 nodeName:}" failed. No retries permitted until 2026-07-26 15:10:12.837359249 +0000 UTC m=+286.735034223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jwj6g" (UID: "62666105-e03c-446e-87dc-06c069722fa9") : secret "insights-runtime-extractor-tls" not found Jul 26 15:09:56.837756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:09:56.837687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62666105-e03c-446e-87dc-06c069722fa9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:10:01.548475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:01.548447 2575 scope.go:117] "RemoveContainer" containerID="30f3b2428a122c7b8ed1988fa3b84b6cea172005f113394c398983bd57b78b87" Jul 26 15:10:02.530746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:02.530717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:10:02.530913 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:02.530778 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" event={"ID":"715bac81-2670-4893-ba37-c3388bacf589","Type":"ContainerStarted","Data":"37699e150561eab93d031fd49d80c8d5588cad27c72f5a25ac5e9c185d4ee6c5"} Jul 26 15:10:02.531043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:02.531024 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:10:02.535511 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:02.535487 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" Jul 26 15:10:02.546595 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:02.546543 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-n9tjl" podStartSLOduration=24.652421217 podStartE2EDuration="26.546525488s" totalImagesPullingTime="1.894104271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:02.545626533 +0000 UTC m=+276.443301530" watchObservedRunningTime="2026-07-26 15:10:02.546525488 +0000 UTC m=+276.444200464" Jul 26 15:10:03.622012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.621971 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-vbg2b"] Jul 26 15:10:03.625705 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.625688 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:03.628095 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.628076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nmsrz\"" Jul 26 15:10:03.628215 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.628131 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 15:10:03.628298 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.628233 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 15:10:03.643717 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.643687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vbg2b"] Jul 26 15:10:03.690634 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.690595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r229s\" (UniqueName: \"kubernetes.io/projected/fde2ca9c-4df1-4964-845f-86430deabdeb-kube-api-access-r229s\") pod \"downloads-7c97dc4555-vbg2b\" (UID: \"fde2ca9c-4df1-4964-845f-86430deabdeb\") " pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:03.733114 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.733073 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f56978d9b-hcfj8"] Jul 26 15:10:03.736838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.736821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.739558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.739534 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 26 15:10:03.739696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.739577 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6qhrv\"" Jul 26 15:10:03.739696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.739535 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 26 15:10:03.739696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.739584 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 26 15:10:03.745172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.745152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 26 15:10:03.747923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.747900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f56978d9b-hcfj8"] Jul 26 15:10:03.791423 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbn9c\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-kube-api-access-zbn9c\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791423 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-certificates\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-bound-sa-token\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-installation-pull-secrets\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-trusted-ca\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-tls\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b012874-44e8-4a11-982b-02bed9f1bbcc-ca-trust-extracted\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.791807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r229s\" (UniqueName: \"kubernetes.io/projected/fde2ca9c-4df1-4964-845f-86430deabdeb-kube-api-access-r229s\") pod \"downloads-7c97dc4555-vbg2b\" (UID: \"fde2ca9c-4df1-4964-845f-86430deabdeb\") " pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:03.791807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.791703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-image-registry-private-configuration\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.802688 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.802655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r229s\" (UniqueName: \"kubernetes.io/projected/fde2ca9c-4df1-4964-845f-86430deabdeb-kube-api-access-r229s\") pod \"downloads-7c97dc4555-vbg2b\" (UID: \"fde2ca9c-4df1-4964-845f-86430deabdeb\") " pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:03.892924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.892840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-tls\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.892924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.892893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b012874-44e8-4a11-982b-02bed9f1bbcc-ca-trust-extracted\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.892924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.892915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-image-registry-private-configuration\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbn9c\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-kube-api-access-zbn9c\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-certificates\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-bound-sa-token\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-installation-pull-secrets\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-trusted-ca\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.893439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b012874-44e8-4a11-982b-02bed9f1bbcc-ca-trust-extracted\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.894007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.893986 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-certificates\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.894357 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.894330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b012874-44e8-4a11-982b-02bed9f1bbcc-trusted-ca\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.895566 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.895546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-image-registry-private-configuration\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.895778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.895760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-registry-tls\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.895821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.895772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b012874-44e8-4a11-982b-02bed9f1bbcc-installation-pull-secrets\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.901562 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.901538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-bound-sa-token\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.901730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.901715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbn9c\" (UniqueName: \"kubernetes.io/projected/7b012874-44e8-4a11-982b-02bed9f1bbcc-kube-api-access-zbn9c\") pod \"image-registry-7f56978d9b-hcfj8\" (UID: \"7b012874-44e8-4a11-982b-02bed9f1bbcc\") " pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:03.933908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:03.933868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:04.047720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.047683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:04.072497 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.072467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-vbg2b"] Jul 26 15:10:04.075170 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:04.075140 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde2ca9c_4df1_4964_845f_86430deabdeb.slice/crio-dd70a014200d25d9909663d4663bfd8e4c3d9ac4d9dfab71c3d966eecfb16c90 WatchSource:0}: Error finding container dd70a014200d25d9909663d4663bfd8e4c3d9ac4d9dfab71c3d966eecfb16c90: Status 404 returned error can't find the container with id dd70a014200d25d9909663d4663bfd8e4c3d9ac4d9dfab71c3d966eecfb16c90 Jul 26 15:10:04.179950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.179918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f56978d9b-hcfj8"] Jul 26 15:10:04.183471 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:04.183445 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b012874_44e8_4a11_982b_02bed9f1bbcc.slice/crio-98237ed17057ec7247564c9a3c146578b1818ba9980fee86e3bdea886b03cbcd WatchSource:0}: Error finding container 98237ed17057ec7247564c9a3c146578b1818ba9980fee86e3bdea886b03cbcd: Status 404 returned error can't find the container with id 98237ed17057ec7247564c9a3c146578b1818ba9980fee86e3bdea886b03cbcd Jul 26 15:10:04.538594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.538477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" event={"ID":"7b012874-44e8-4a11-982b-02bed9f1bbcc","Type":"ContainerStarted","Data":"688769d8cf15559621648460de703ef4d2afc4ef44ff16721f8cb274a9775895"} Jul 26 15:10:04.538594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.538555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" event={"ID":"7b012874-44e8-4a11-982b-02bed9f1bbcc","Type":"ContainerStarted","Data":"98237ed17057ec7247564c9a3c146578b1818ba9980fee86e3bdea886b03cbcd"} Jul 26 15:10:04.538594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.538590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:04.539563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.539540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vbg2b" event={"ID":"fde2ca9c-4df1-4964-845f-86430deabdeb","Type":"ContainerStarted","Data":"dd70a014200d25d9909663d4663bfd8e4c3d9ac4d9dfab71c3d966eecfb16c90"} Jul 26 15:10:04.556199 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:04.556151 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" podStartSLOduration=1.556135075 podStartE2EDuration="1.556135075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:04.555675684 +0000 UTC m=+278.453350681" watchObservedRunningTime="2026-07-26 15:10:04.556135075 +0000 UTC m=+278.453810071" Jul 26 15:10:07.198160 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:10:07.198106 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z4s5l" podUID="96663026-7fcd-4267-a3f8-988a24bacacc" Jul 26 15:10:07.549236 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:07.549152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:10.451121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:10.451081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:10.453791 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:10.453767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96663026-7fcd-4267-a3f8-988a24bacacc-metrics-tls\") pod \"dns-default-z4s5l\" (UID: \"96663026-7fcd-4267-a3f8-988a24bacacc\") " pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:10.552996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:10.552962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8qc59\"" Jul 26 15:10:10.560693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:10.560661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:10.706733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:10.706570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4s5l"] Jul 26 15:10:10.710372 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:10.710330 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96663026_7fcd_4267_a3f8_988a24bacacc.slice/crio-5e472023c10491a369825f99bee74f45f4d4a4f7f86a59579b425f56ce11d96d WatchSource:0}: Error finding container 5e472023c10491a369825f99bee74f45f4d4a4f7f86a59579b425f56ce11d96d: Status 404 returned error can't find the container with id 5e472023c10491a369825f99bee74f45f4d4a4f7f86a59579b425f56ce11d96d Jul 26 15:10:11.563682 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:11.563601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4s5l" event={"ID":"96663026-7fcd-4267-a3f8-988a24bacacc","Type":"ContainerStarted","Data":"5e472023c10491a369825f99bee74f45f4d4a4f7f86a59579b425f56ce11d96d"} Jul 26 15:10:12.873546 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:12.873433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:10:12.876412 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:12.876378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62666105-e03c-446e-87dc-06c069722fa9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jwj6g\" (UID: \"62666105-e03c-446e-87dc-06c069722fa9\") " pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:10:13.119936 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.119887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jwj6g" Jul 26 15:10:13.271715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.271678 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jwj6g"] Jul 26 15:10:13.276066 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:13.276039 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62666105_e03c_446e_87dc_06c069722fa9.slice/crio-d6ae0aa8379a90fe2345c58f558039a95bf173c066e13fcc19bc71836d35c6e4 WatchSource:0}: Error finding container d6ae0aa8379a90fe2345c58f558039a95bf173c066e13fcc19bc71836d35c6e4: Status 404 returned error can't find the container with id d6ae0aa8379a90fe2345c58f558039a95bf173c066e13fcc19bc71836d35c6e4 Jul 26 15:10:13.573415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.573089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4s5l" event={"ID":"96663026-7fcd-4267-a3f8-988a24bacacc","Type":"ContainerStarted","Data":"5b62526b09d7bba370d7f68acc28fe389634c2ae79dc1936ad6b1bb31ce0ee7e"} Jul 26 15:10:13.573415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.573155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4s5l" event={"ID":"96663026-7fcd-4267-a3f8-988a24bacacc","Type":"ContainerStarted","Data":"cf29f2552b6288214c2cf3cf1da6f2ac92465e332cdd517b2219bf8a56f6a928"} Jul 26 15:10:13.573415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.573203 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:13.574420 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.574391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwj6g" event={"ID":"62666105-e03c-446e-87dc-06c069722fa9","Type":"ContainerStarted","Data":"d6ae0aa8379a90fe2345c58f558039a95bf173c066e13fcc19bc71836d35c6e4"} Jul 26 15:10:13.590176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:13.590123 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z4s5l" podStartSLOduration=251.737959356 podStartE2EDuration="4m13.590106419s" totalImagesPullingTime="1.852147063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:13.588946296 +0000 UTC m=+287.486621293" watchObservedRunningTime="2026-07-26 15:10:13.590106419 +0000 UTC m=+287.487781416" Jul 26 15:10:19.550625 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:10:19.550576 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5ts4r" podUID="89b1a676-2fb9-4294-8a5a-6fd4e0c83da2" Jul 26 15:10:20.678538 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.678492 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv"] Jul 26 15:10:20.681876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.681847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.686008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.685970 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nlfkn\"" Jul 26 15:10:20.686136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686039 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 15:10:20.686194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686044 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 26 15:10:20.686194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686159 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 15:10:20.686194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686171 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 26 15:10:20.686321 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686233 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 15:10:20.686361 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.686319 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 26 15:10:20.694650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.694621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv"] Jul 26 15:10:20.697572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.697543 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2b97j"] Jul 26 15:10:20.700743 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.700718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.703256 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.703230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 15:10:20.703256 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.703234 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 15:10:20.703704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.703612 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 15:10:20.703704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.703646 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gn4ps\"" Jul 26 15:10:20.738964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.738920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.738964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.738970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.739209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.739016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.739209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.739051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwzz\" (UniqueName: \"kubernetes.io/projected/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-api-access-4hwzz\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.739209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.739110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.739209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.739167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840343 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-metrics-client-ca\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-root\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwzz\" (UniqueName: \"kubernetes.io/projected/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-api-access-4hwzz\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-textfile\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-sys\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-wtmp\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-tls\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2kb\" (UniqueName: \"kubernetes.io/projected/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-kube-api-access-mz2kb\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.840843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.840829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.841334 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.841308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.841633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.841612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.843463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.843441 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.844169 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.844145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.849198 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.849174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwzz\" (UniqueName: \"kubernetes.io/projected/3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6-kube-api-access-4hwzz\") pod \"kube-state-metrics-56d9f944b4-shtnv\" (UID: \"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:20.942132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-metrics-client-ca\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-root\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-textfile\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-sys\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-wtmp\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-tls\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2kb\" (UniqueName: \"kubernetes.io/projected/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-kube-api-access-mz2kb\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-root\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942803 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942869 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942820 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-sys\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942869 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942854 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-wtmp\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-textfile\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.942964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.942914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-metrics-client-ca\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.944838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.944814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.945221 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.945200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-node-exporter-tls\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.951722 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.951698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2kb\" (UniqueName: \"kubernetes.io/projected/59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e-kube-api-access-mz2kb\") pod \"node-exporter-2b97j\" (UID: \"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e\") " pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:20.993962 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:20.993926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" Jul 26 15:10:21.011085 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.011059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2b97j" Jul 26 15:10:21.745909 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.745868 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:10:21.757230 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.757196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.759905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.759869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 26 15:10:21.760049 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.759914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 26 15:10:21.760049 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.759952 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 26 15:10:21.760165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760077 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9l9ww\"" Jul 26 15:10:21.760267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760246 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 26 15:10:21.760483 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 26 15:10:21.760555 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760538 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 26 15:10:21.760789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760636 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 26 15:10:21.760789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760694 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 26 15:10:21.760789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.760715 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 26 15:10:21.766386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.766364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:10:21.849085 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbndx\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.849789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.849553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950420 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950420 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbndx\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.950689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.950847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.951631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.951591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.952337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.951816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.952337 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:10:21.951920 2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 26 15:10:21.952337 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:10:21.951999 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls podName:a627cd7a-7036-4b29-920e-fc825f189749 nodeName:}" failed. No retries permitted until 2026-07-26 15:10:22.451979467 +0000 UTC m=+296.349654446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a627cd7a-7036-4b29-920e-fc825f189749") : secret "alertmanager-main-tls" not found Jul 26 15:10:21.954295 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.954247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.954825 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.954707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.954825 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.954791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.954972 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.954922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.955351 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.955329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.955902 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.955877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.956326 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.956310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.956876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.956856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:21.959821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:21.959803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbndx\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:22.455795 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:22.455746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:22.458631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:22.458590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:22.669412 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:22.669368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:10:23.581962 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:23.581932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4s5l" Jul 26 15:10:24.222133 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:24.221864 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ea7772_ae1f_4c28_8c64_20e3f9c5ba8e.slice/crio-0347394494d3389d8059b762a5345124fbf86b1c2dc74635f1fbaf5c2b67e441 WatchSource:0}: Error finding container 0347394494d3389d8059b762a5345124fbf86b1c2dc74635f1fbaf5c2b67e441: Status 404 returned error can't find the container with id 0347394494d3389d8059b762a5345124fbf86b1c2dc74635f1fbaf5c2b67e441 Jul 26 15:10:24.380131 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.380098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv"] Jul 26 15:10:24.383156 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:24.383127 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc2fd4a_7d0c_4c5c_a333_adac8a9c31e6.slice/crio-5eb911f5a1faefff2463f9b40eddf55982dfbbdafc6ee8601736f288197c3f05 WatchSource:0}: Error finding container 5eb911f5a1faefff2463f9b40eddf55982dfbbdafc6ee8601736f288197c3f05: Status 404 returned error can't find the container with id 5eb911f5a1faefff2463f9b40eddf55982dfbbdafc6ee8601736f288197c3f05 Jul 26 15:10:24.411844 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.411811 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:10:24.414791 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:10:24.414765 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda627cd7a_7036_4b29_920e_fc825f189749.slice/crio-3197122522ecb9da52640ac8de1d802412f648de0b587b8af052a5563011abbe WatchSource:0}: Error finding container 3197122522ecb9da52640ac8de1d802412f648de0b587b8af052a5563011abbe: Status 404 returned error can't find the container with id 3197122522ecb9da52640ac8de1d802412f648de0b587b8af052a5563011abbe Jul 26 15:10:24.621864 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.621776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b97j" event={"ID":"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e","Type":"ContainerStarted","Data":"0347394494d3389d8059b762a5345124fbf86b1c2dc74635f1fbaf5c2b67e441"} Jul 26 15:10:24.623653 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.623578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwj6g" event={"ID":"62666105-e03c-446e-87dc-06c069722fa9","Type":"ContainerStarted","Data":"32edec6df09bda250127992b315690ff8e6908a015868fbcf17f223ebc789a86"} Jul 26 15:10:24.624851 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.624820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"3197122522ecb9da52640ac8de1d802412f648de0b587b8af052a5563011abbe"} Jul 26 15:10:24.626454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.626427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-vbg2b" event={"ID":"fde2ca9c-4df1-4964-845f-86430deabdeb","Type":"ContainerStarted","Data":"31718ad5871fc04ce14e122966ade1654af1f0f9d0c3b1283de14dd92df3e55c"} Jul 26 15:10:24.626932 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.626909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:24.628126 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.628100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" event={"ID":"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6","Type":"ContainerStarted","Data":"5eb911f5a1faefff2463f9b40eddf55982dfbbdafc6ee8601736f288197c3f05"} Jul 26 15:10:24.642117 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.642076 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-vbg2b" Jul 26 15:10:24.643992 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:24.643941 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-vbg2b" podStartSLOduration=1.5012027909999999 podStartE2EDuration="21.643925284s" totalImagesPullingTime="20.142722493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:24.642417541 +0000 UTC m=+298.540092538" watchObservedRunningTime="2026-07-26 15:10:24.643925284 +0000 UTC m=+298.541600258" Jul 26 15:10:25.549256 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:25.549226 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f56978d9b-hcfj8" Jul 26 15:10:30.298939 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.298907 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:10:30.299399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.299145 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:10:30.311674 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.311648 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 15:10:30.552331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.552221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:10:30.668847 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.668813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" event={"ID":"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6","Type":"ContainerStarted","Data":"3ba27204e6950bc9700bdb279a4a6984ee63f7021453daedceb05db2ec280ea5"} Jul 26 15:10:30.668991 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.668886 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" event={"ID":"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6","Type":"ContainerStarted","Data":"a34ba79827f6c818915a1921a44c8f393e594d06155dc6c4ca82fa13f9698d3b"} Jul 26 15:10:30.670880 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.670853 2575 generic.go:356] "Generic (PLEG): container finished" podID="59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e" containerID="84e1686807e26731b412c432eaed73e0f284ed97e283bcea34c67e773f93dee8" exitCode=0 Jul 26 15:10:30.671013 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.670962 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b97j" event={"ID":"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e","Type":"ContainerDied","Data":"84e1686807e26731b412c432eaed73e0f284ed97e283bcea34c67e773f93dee8"} Jul 26 15:10:30.673001 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.672974 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" exitCode=0 Jul 26 15:10:30.673111 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.673013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} Jul 26 15:10:30.674425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:30.674407 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:10:31.683434 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.683238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwj6g" event={"ID":"62666105-e03c-446e-87dc-06c069722fa9","Type":"ContainerStarted","Data":"599a48c43d49380a5de39585c26f8638bcc6d78d7a0e1a62f5bf24a4a42e9a3c"} Jul 26 15:10:31.690759 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.689965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" event={"ID":"3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6","Type":"ContainerStarted","Data":"b096bafc609cf5ec9a78991f1bc5f3f5038970e043bf923979aa16aa80e1aac3"} Jul 26 15:10:31.693691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.693606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b97j" event={"ID":"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e","Type":"ContainerStarted","Data":"a8336b57fec46e79f3cdf8437ec971808969a7ab07c0b483299d5447876b52a6"} Jul 26 15:10:31.693831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.693693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2b97j" event={"ID":"59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e","Type":"ContainerStarted","Data":"5eac055c8588d61b09d51fae78211f5655588e074f409a2eceb619c62990caac"} Jul 26 15:10:31.711307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.711220 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-shtnv" podStartSLOduration=5.809914389 podStartE2EDuration="11.711199424s" totalImagesPullingTime="5.901285035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:31.708978261 +0000 UTC m=+305.606653260" watchObservedRunningTime="2026-07-26 15:10:31.711199424 +0000 UTC m=+305.608874423" Jul 26 15:10:31.729098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:31.727362 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2b97j" podStartSLOduration=5.514401996 podStartE2EDuration="11.727339275s" totalImagesPullingTime="6.062824362s" totalInitContainerRuntime="150.112917ms" isStatefulPod=true podCreationTimestamp="2026-07-26 15:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:31.726949455 +0000 UTC m=+305.624624452" watchObservedRunningTime="2026-07-26 15:10:31.727339275 +0000 UTC m=+305.625014275" Jul 26 15:10:35.712232 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:35.712200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} Jul 26 15:10:35.712593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:35.712298 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} Jul 26 15:10:35.714016 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:35.713991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jwj6g" event={"ID":"62666105-e03c-446e-87dc-06c069722fa9","Type":"ContainerStarted","Data":"c1b4f3f85dc3521c8001ec6c4e231efcf510ee94fd99492653222f97e300119a"} Jul 26 15:10:35.732861 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:35.732808 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jwj6g" podStartSLOduration=33.670765589 podStartE2EDuration="55.732791761s" totalImagesPullingTime="22.062026172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:09:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:35.73106446 +0000 UTC m=+309.628739453" watchObservedRunningTime="2026-07-26 15:10:35.732791761 +0000 UTC m=+309.630466757" Jul 26 15:10:36.720538 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:36.720500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} Jul 26 15:10:36.721108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:36.720583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} Jul 26 15:10:36.721108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:36.720660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} Jul 26 15:10:37.727864 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:37.727826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerStarted","Data":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} Jul 26 15:10:37.755416 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:37.755117 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.162277939 podStartE2EDuration="16.755095568s" totalImagesPullingTime="11.560865793s" totalInitContainerRuntime="31.951836ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:10:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:10:37.752995625 +0000 UTC m=+311.650670622" watchObservedRunningTime="2026-07-26 15:10:37.755095568 +0000 UTC m=+311.652770566" Jul 26 15:10:45.781201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:45.781145 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" podUID="88a1cfa1-33ce-48a3-87b5-f10f88373f78" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 26 15:10:55.780913 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:10:55.780866 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" podUID="88a1cfa1-33ce-48a3-87b5-f10f88373f78" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 26 15:11:05.780668 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:05.780617 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" podUID="88a1cfa1-33ce-48a3-87b5-f10f88373f78" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 26 15:11:05.781120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:05.780707 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" Jul 26 15:11:05.781191 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:05.781158 2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d1ada27ce8bbcd15912b4f8237137ced307d99cc91ba5af2538ee3df9f0d2a70"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 26 15:11:05.781239 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:05.781212 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" podUID="88a1cfa1-33ce-48a3-87b5-f10f88373f78" containerName="service-proxy" containerID="cri-o://d1ada27ce8bbcd15912b4f8237137ced307d99cc91ba5af2538ee3df9f0d2a70" gracePeriod=30 Jul 26 15:11:06.829349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:06.829317 2575 generic.go:356] "Generic (PLEG): container finished" podID="88a1cfa1-33ce-48a3-87b5-f10f88373f78" containerID="d1ada27ce8bbcd15912b4f8237137ced307d99cc91ba5af2538ee3df9f0d2a70" exitCode=2 Jul 26 15:11:06.829865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:06.829366 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerDied","Data":"d1ada27ce8bbcd15912b4f8237137ced307d99cc91ba5af2538ee3df9f0d2a70"} Jul 26 15:11:06.829865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:06.829409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f86c557f7-c9h2q" event={"ID":"88a1cfa1-33ce-48a3-87b5-f10f88373f78","Type":"ContainerStarted","Data":"d26bbf83376b5da59589dfbcb03ad82c1e01a7b60b9867a3997e48135fd862fb"} Jul 26 15:11:39.413032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.412991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:11:39.415448 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.415424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89b1a676-2fb9-4294-8a5a-6fd4e0c83da2-metrics-certs\") pod \"network-metrics-daemon-5ts4r\" (UID: \"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2\") " pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:11:39.556096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.556062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgnb2\"" Jul 26 15:11:39.564156 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.564133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ts4r" Jul 26 15:11:39.690947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.690923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ts4r"] Jul 26 15:11:39.693468 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:11:39.693439 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b1a676_2fb9_4294_8a5a_6fd4e0c83da2.slice/crio-284772717a6967d467e859fd2bbfa1bd4e16870f7c7e67505925aa27660c6abf WatchSource:0}: Error finding container 284772717a6967d467e859fd2bbfa1bd4e16870f7c7e67505925aa27660c6abf: Status 404 returned error can't find the container with id 284772717a6967d467e859fd2bbfa1bd4e16870f7c7e67505925aa27660c6abf Jul 26 15:11:39.945270 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:39.945177 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ts4r" event={"ID":"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2","Type":"ContainerStarted","Data":"284772717a6967d467e859fd2bbfa1bd4e16870f7c7e67505925aa27660c6abf"} Jul 26 15:11:40.950679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.950637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ts4r" event={"ID":"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2","Type":"ContainerStarted","Data":"d2e8a1e443cf56478682b729685fd19357f40cace75960d97f40cf41a7b0a229"} Jul 26 15:11:40.951054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.950690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ts4r" event={"ID":"89b1a676-2fb9-4294-8a5a-6fd4e0c83da2","Type":"ContainerStarted","Data":"92414740b66d881584914727332b9e84fe011e13633264c44be4267e0bd178af"} Jul 26 15:11:40.967431 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.967359 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5ts4r" podStartSLOduration=374.155920767 podStartE2EDuration="6m14.967341435s" totalImagesPullingTime="811.420668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:11:40.966124322 +0000 UTC m=+374.863799318" watchObservedRunningTime="2026-07-26 15:11:40.967341435 +0000 UTC m=+374.865016431" Jul 26 15:11:40.991639 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.991608 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:40.991980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.991922 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="alertmanager" containerID="cri-o://315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" gracePeriod=120 Jul 26 15:11:40.992109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.991980 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-metric" containerID="cri-o://92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" gracePeriod=120 Jul 26 15:11:40.992109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.991989 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="config-reloader" containerID="cri-o://8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" gracePeriod=120 Jul 26 15:11:40.992109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.992022 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy" containerID="cri-o://777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" gracePeriod=120 Jul 26 15:11:40.992109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.991980 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-web" containerID="cri-o://1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" gracePeriod=120 Jul 26 15:11:40.992109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:40.992058 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="prom-label-proxy" containerID="cri-o://a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" gracePeriod=120 Jul 26 15:11:41.234833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.234807 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:41.326386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbndx\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326389 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326425 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326445 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326472 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326490 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326531 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326566 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.326621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.327027 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326635 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.327027 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.327027 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.326694 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca\") pod \"a627cd7a-7036-4b29-920e-fc825f189749\" (UID: \"a627cd7a-7036-4b29-920e-fc825f189749\") " Jul 26 15:11:41.327301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.327154 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:41.327712 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.327483 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:11:41.327828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.327758 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:11:41.329598 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.329573 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.329991 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.329761 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:11:41.331556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx" (OuterVolumeSpecName: "kube-api-access-bbndx") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "kube-api-access-bbndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:11:41.331640 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331551 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out" (OuterVolumeSpecName: "config-out") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:11:41.331640 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331560 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.331640 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume" (OuterVolumeSpecName: "config-volume") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.331778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331692 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.331778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.331749 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.335166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.335146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.341440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.341415 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config" (OuterVolumeSpecName: "web-config") pod "a627cd7a-7036-4b29-920e-fc825f189749" (UID: "a627cd7a-7036-4b29-920e-fc825f189749"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:11:41.427529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427481 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427525 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-tls-assets\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427536 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427546 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-web-config\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427555 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427564 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427573 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427582 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-config-out\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427590 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a627cd7a-7036-4b29-920e-fc825f189749-alertmanager-main-db\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427599 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-cluster-tls-config\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427606 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a627cd7a-7036-4b29-920e-fc825f189749-metrics-client-ca\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427616 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbndx\" (UniqueName: \"kubernetes.io/projected/a627cd7a-7036-4b29-920e-fc825f189749-kube-api-access-bbndx\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.427796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.427624 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a627cd7a-7036-4b29-920e-fc825f189749-secret-alertmanager-main-tls\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:11:41.957012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.956980 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" exitCode=0 Jul 26 15:11:41.957012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957007 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" exitCode=0 Jul 26 15:11:41.957012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957014 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" exitCode=0 Jul 26 15:11:41.957012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957020 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" exitCode=0 Jul 26 15:11:41.957012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957024 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" exitCode=0 Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957030 2575 generic.go:356] "Generic (PLEG): container finished" podID="a627cd7a-7036-4b29-920e-fc825f189749" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" exitCode=0 Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957069 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957168 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957159 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} Jul 26 15:11:41.957718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.957406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a627cd7a-7036-4b29-920e-fc825f189749","Type":"ContainerDied","Data":"3197122522ecb9da52640ac8de1d802412f648de0b587b8af052a5563011abbe"} Jul 26 15:11:41.965623 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.965605 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:41.972780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.972759 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:41.980400 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.980372 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:41.981525 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.981487 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:41.986369 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.986342 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:41.989816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.989793 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:41.996958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:41.996940 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.004480 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.004457 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.011748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.011726 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.012021 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.012000 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.012070 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012033 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.012070 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012052 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.012319 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.012298 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.012372 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012328 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.012372 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012346 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.012582 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.012565 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.012628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012586 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.012628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012600 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.012792 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.012777 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.012831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012795 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.012831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.012809 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.013045 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.013029 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.013088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013048 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.013088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013061 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.013301 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.013266 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.013346 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013307 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.013346 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013323 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.013539 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:11:42.013523 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.013578 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013543 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.013578 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013556 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.013775 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013758 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.013817 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013776 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.013982 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013966 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.014069 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.013983 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.014189 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014173 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.014228 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014190 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.014495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014475 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.014550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014497 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.014768 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014751 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.014812 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014768 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.014977 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014960 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.014977 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.014977 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.015210 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015191 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.015288 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015211 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.015498 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015479 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.015541 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015499 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.015866 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015709 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.015866 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.015737 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.016116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016059 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.016116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016083 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.016481 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016402 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.016481 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016440 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.016744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016705 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.016794 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.016746 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.017079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017057 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.017079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017080 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.017373 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017351 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.017442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017376 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.017628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017600 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.017628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017626 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.017905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017883 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.017905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.017906 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.018122 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018104 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.018167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018120 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:42.018207 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018122 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.018399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018385 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-web" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018400 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-web" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="prom-label-proxy" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018419 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="prom-label-proxy" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018436 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="alertmanager" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018443 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="alertmanager" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018451 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy" Jul 26 15:11:42.018455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018463 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="init-config-reloader" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018469 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="init-config-reloader" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018477 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-metric" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018482 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-metric" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018499 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="config-reloader" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018507 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="config-reloader" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018557 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="prom-label-proxy" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018567 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-web" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018572 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="alertmanager" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018577 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018583 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="kube-rbac-proxy-metric" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018428 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018603 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.018680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018588 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a627cd7a-7036-4b29-920e-fc825f189749" containerName="config-reloader" Jul 26 15:11:42.019103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018836 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.019103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.018865 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.019103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.019071 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.019103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.019090 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.024063 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024031 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.024063 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024062 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.024372 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024343 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.024372 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024365 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.024617 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024592 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.024672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024621 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.024860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024832 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.024860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.024860 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.025090 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025073 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.025090 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025091 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.025384 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025350 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.025384 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025383 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.025654 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025627 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.025720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025660 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.025843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.025944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025920 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.026004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.025950 2575 scope.go:117] "RemoveContainer" containerID="a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54" Jul 26 15:11:42.026390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.026358 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54"} err="failed to get container status \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": rpc error: code = NotFound desc = could not find container \"a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54\": container with ID starting with a73760dea41f44e3a2cb752170f43f3d540bfbab3ddf631b548ae0e20b473a54 not found: ID does not exist" Jul 26 15:11:42.026471 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.026398 2575 scope.go:117] "RemoveContainer" containerID="92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd" Jul 26 15:11:42.026728 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.026708 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd"} err="failed to get container status \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": rpc error: code = NotFound desc = could not find container \"92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd\": container with ID starting with 92fac428b77b64a01a48587021a5e51a4e7a253a0c25da210cc77cd320d646fd not found: ID does not exist" Jul 26 15:11:42.026786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.026729 2575 scope.go:117] "RemoveContainer" containerID="777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462" Jul 26 15:11:42.026999 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.026975 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462"} err="failed to get container status \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": rpc error: code = NotFound desc = could not find container \"777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462\": container with ID starting with 777f4cd7fd5078e43db915e89def5a1592ad5d7f2cc44b048efea5e46a947462 not found: ID does not exist" Jul 26 15:11:42.027071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027001 2575 scope.go:117] "RemoveContainer" containerID="1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3" Jul 26 15:11:42.027287 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027248 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3"} err="failed to get container status \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": rpc error: code = NotFound desc = could not find container \"1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3\": container with ID starting with 1baca81d4704336716f99cacbb5c1b9c7d3297df3cb39bda70fec99aa69d26e3 not found: ID does not exist" Jul 26 15:11:42.027349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027299 2575 scope.go:117] "RemoveContainer" containerID="8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef" Jul 26 15:11:42.027714 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027686 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef"} err="failed to get container status \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": rpc error: code = NotFound desc = could not find container \"8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef\": container with ID starting with 8d5ac80d5d1882b2b137de9afdbbb16ae8401f9e7083ad923ea2695738c124ef not found: ID does not exist" Jul 26 15:11:42.027792 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027716 2575 scope.go:117] "RemoveContainer" containerID="315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb" Jul 26 15:11:42.027991 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027973 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb"} err="failed to get container status \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": rpc error: code = NotFound desc = could not find container \"315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb\": container with ID starting with 315c9b592467feded17764d673723570e3e59c308e35e796547a709b4f8993fb not found: ID does not exist" Jul 26 15:11:42.028046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.027993 2575 scope.go:117] "RemoveContainer" containerID="b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07" Jul 26 15:11:42.028242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.028225 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07"} err="failed to get container status \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": rpc error: code = NotFound desc = could not find container \"b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07\": container with ID starting with b407b0ac57a1e664f220ac0ce2d78e0e0b0ece61d59ec19628e8a9f9d5211a07 not found: ID does not exist" Jul 26 15:11:42.028639 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.028625 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 26 15:11:42.028697 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.028627 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 26 15:11:42.029043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029026 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 26 15:11:42.029115 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 26 15:11:42.029329 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029309 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 26 15:11:42.029410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 26 15:11:42.029410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 26 15:11:42.029410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029381 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 26 15:11:42.029657 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.029643 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9l9ww\"" Jul 26 15:11:42.031635 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-web-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-config-out\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031894 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031894 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.031894 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-config-volume\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032042 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032042 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032042 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.031988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032042 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.032008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032042 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.032023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffwp\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-kube-api-access-2ffwp\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.032306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.032102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.035114 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.035095 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 26 15:11:42.039622 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.039599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:42.133237 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133237 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-web-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-config-out\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-config-volume\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffwp\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-kube-api-access-2ffwp\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.133959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.133810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.134425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.134397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.135573 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.135184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926ce04b-980e-447c-b385-a84b190b9cab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.136267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.136227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-web-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.136832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.136806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.136981 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.136954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.137076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.137049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.137187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.136971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-config-volume\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.137831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.137811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/926ce04b-980e-447c-b385-a84b190b9cab-config-out\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.138002 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.137982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.138109 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.138088 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.138333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.138316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/926ce04b-980e-447c-b385-a84b190b9cab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.145442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.145420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffwp\" (UniqueName: \"kubernetes.io/projected/926ce04b-980e-447c-b385-a84b190b9cab-kube-api-access-2ffwp\") pod \"alertmanager-main-0\" (UID: \"926ce04b-980e-447c-b385-a84b190b9cab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.337019 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.336923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 15:11:42.473212 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.473185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 15:11:42.475172 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:11:42.475147 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926ce04b_980e_447c_b385_a84b190b9cab.slice/crio-5ba3e2b018402f2412f34d460601597a238573568507982a643cb63ae1f68bc7 WatchSource:0}: Error finding container 5ba3e2b018402f2412f34d460601597a238573568507982a643cb63ae1f68bc7: Status 404 returned error can't find the container with id 5ba3e2b018402f2412f34d460601597a238573568507982a643cb63ae1f68bc7 Jul 26 15:11:42.554099 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.554076 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a627cd7a-7036-4b29-920e-fc825f189749" path="/var/lib/kubelet/pods/a627cd7a-7036-4b29-920e-fc825f189749/volumes" Jul 26 15:11:42.963551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.963516 2575 generic.go:356] "Generic (PLEG): container finished" podID="926ce04b-980e-447c-b385-a84b190b9cab" containerID="8c3ed7cc1963fe93fef5ad71e85c75b067d2465133c359c9a23ade808cda63bd" exitCode=0 Jul 26 15:11:42.963971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.963557 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerDied","Data":"8c3ed7cc1963fe93fef5ad71e85c75b067d2465133c359c9a23ade808cda63bd"} Jul 26 15:11:42.963971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:42.963607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"5ba3e2b018402f2412f34d460601597a238573568507982a643cb63ae1f68bc7"} Jul 26 15:11:43.971030 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.970991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"1ade598182f9adc2ea7a20a50c01d853644243986e408daa7f9b2b879978a729"} Jul 26 15:11:43.971504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.971063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"556b10f46a734f3fde26b5ae254f868f87fee9f791461756df78d5a158d17d92"} Jul 26 15:11:43.971504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.971092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"70a5dccc7c7802604fdb9c74f6225ba31dcd897f140de43928e6e37ea497d9a4"} Jul 26 15:11:43.971504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.971127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"e20bb5ecb1b0bb25f45f3417cfabb13c298626df947efff06a413a27aa209400"} Jul 26 15:11:43.971504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.971157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"6a323571114835540d124dd965a5282fd91d20632f0fd505889dbdc76533f742"} Jul 26 15:11:43.971504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:43.971182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"926ce04b-980e-447c-b385-a84b190b9cab","Type":"ContainerStarted","Data":"7a8895f13d81927c6921fd67f2ce422ac8629d329adaa70e39236758beadee1a"} Jul 26 15:11:44.005820 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:11:44.005755 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.98938322 podStartE2EDuration="3.005733216s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.349996ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:11:44.004976596 +0000 UTC m=+377.902651592" watchObservedRunningTime="2026-07-26 15:11:44.005733216 +0000 UTC m=+377.903408229" Jul 26 15:12:28.295572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.295532 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j"] Jul 26 15:12:28.298530 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.298506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.301318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.301270 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:12:28.301318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.301305 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:12:28.301495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.301347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5thlm\"" Jul 26 15:12:28.307244 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.307076 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j"] Jul 26 15:12:28.389488 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.389454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.389671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.389511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dkm7\" (UniqueName: \"kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.389671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.389589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.490558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.490508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.490730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.490575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dkm7\" (UniqueName: \"kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.490730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.490606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.490935 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.490915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.490972 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.490935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.498877 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.498847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dkm7\" (UniqueName: \"kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.607415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.607321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:28.736455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:28.736377 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j"] Jul 26 15:12:28.739020 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:12:28.738987 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615ffbc1_0840_4484_9d1e_d32bde88d900.slice/crio-7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139 WatchSource:0}: Error finding container 7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139: Status 404 returned error can't find the container with id 7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139 Jul 26 15:12:29.128005 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:29.127967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" event={"ID":"615ffbc1-0840-4484-9d1e-d32bde88d900","Type":"ContainerStarted","Data":"7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139"} Jul 26 15:12:35.154267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:35.154229 2575 generic.go:356] "Generic (PLEG): container finished" podID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerID="3f3179092db20f4cef22654021a51c49f8a126070fef0a807460f6504009ddc3" exitCode=0 Jul 26 15:12:35.154267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:35.154299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" event={"ID":"615ffbc1-0840-4484-9d1e-d32bde88d900","Type":"ContainerDied","Data":"3f3179092db20f4cef22654021a51c49f8a126070fef0a807460f6504009ddc3"} Jul 26 15:12:38.167583 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:38.167546 2575 generic.go:356] "Generic (PLEG): container finished" podID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerID="ac012ae994082bd2611b0cf7d3f6970e35443a15c90786249f4fbcf73b1e6dd7" exitCode=0 Jul 26 15:12:38.167990 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:38.167634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" event={"ID":"615ffbc1-0840-4484-9d1e-d32bde88d900","Type":"ContainerDied","Data":"ac012ae994082bd2611b0cf7d3f6970e35443a15c90786249f4fbcf73b1e6dd7"} Jul 26 15:12:45.199406 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:45.199365 2575 generic.go:356] "Generic (PLEG): container finished" podID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerID="60be285482ab45e36dcae21924bcc923223c720c4b37d4db68b71e34c124b01b" exitCode=0 Jul 26 15:12:45.199779 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:45.199443 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" event={"ID":"615ffbc1-0840-4484-9d1e-d32bde88d900","Type":"ContainerDied","Data":"60be285482ab45e36dcae21924bcc923223c720c4b37d4db68b71e34c124b01b"} Jul 26 15:12:46.326234 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.326209 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:46.451840 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.451807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle\") pod \"615ffbc1-0840-4484-9d1e-d32bde88d900\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " Jul 26 15:12:46.452001 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.451901 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util\") pod \"615ffbc1-0840-4484-9d1e-d32bde88d900\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " Jul 26 15:12:46.452001 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.451922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dkm7\" (UniqueName: \"kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7\") pod \"615ffbc1-0840-4484-9d1e-d32bde88d900\" (UID: \"615ffbc1-0840-4484-9d1e-d32bde88d900\") " Jul 26 15:12:46.452458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.452427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle" (OuterVolumeSpecName: "bundle") pod "615ffbc1-0840-4484-9d1e-d32bde88d900" (UID: "615ffbc1-0840-4484-9d1e-d32bde88d900"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:12:46.454214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.454157 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7" (OuterVolumeSpecName: "kube-api-access-7dkm7") pod "615ffbc1-0840-4484-9d1e-d32bde88d900" (UID: "615ffbc1-0840-4484-9d1e-d32bde88d900"). InnerVolumeSpecName "kube-api-access-7dkm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:12:46.457160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.457135 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util" (OuterVolumeSpecName: "util") pod "615ffbc1-0840-4484-9d1e-d32bde88d900" (UID: "615ffbc1-0840-4484-9d1e-d32bde88d900"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:12:46.552500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.552471 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:12:46.552500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.552494 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dkm7\" (UniqueName: \"kubernetes.io/projected/615ffbc1-0840-4484-9d1e-d32bde88d900-kube-api-access-7dkm7\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:12:46.552500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:46.552505 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615ffbc1-0840-4484-9d1e-d32bde88d900-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:12:47.209558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:47.209524 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" Jul 26 15:12:47.209558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:47.209534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qpv9j" event={"ID":"615ffbc1-0840-4484-9d1e-d32bde88d900","Type":"ContainerDied","Data":"7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139"} Jul 26 15:12:47.209749 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:47.209600 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2db4a95552dfd38e7cbd8f764a2a6dedd9452431ccda2fa01f17177ebb3139" Jul 26 15:12:55.384656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384621 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq"] Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384902 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="util" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384912 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="util" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384921 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="extract" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384927 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="extract" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="pull" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="pull" Jul 26 15:12:55.385106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.384993 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="615ffbc1-0840-4484-9d1e-d32bde88d900" containerName="extract" Jul 26 15:12:55.387567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.387550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.390518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.390494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5thlm\"" Jul 26 15:12:55.390762 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.390749 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:12:55.391677 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.391659 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:12:55.403408 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.403384 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq"] Jul 26 15:12:55.428889 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.428856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5gv\" (UniqueName: \"kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.429039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.428900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.429039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.428980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.529847 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.529812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.530015 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.529875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.530015 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.529900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5gv\" (UniqueName: \"kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.530331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.530305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.530376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.530318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.543950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.543914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5gv\" (UniqueName: \"kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.695801 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.695775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:12:55.826878 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:55.826823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq"] Jul 26 15:12:55.830492 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:12:55.830465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd4cbe_067a_491e_9d01_3636490460ff.slice/crio-4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1 WatchSource:0}: Error finding container 4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1: Status 404 returned error can't find the container with id 4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1 Jul 26 15:12:56.248714 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:56.248682 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerID="f1e57298945685ca818d14359322dbc4dd92852cbae4ac9179db62fc6e46c474" exitCode=0 Jul 26 15:12:56.248895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:56.248725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" event={"ID":"c1dd4cbe-067a-491e-9d01-3636490460ff","Type":"ContainerDied","Data":"f1e57298945685ca818d14359322dbc4dd92852cbae4ac9179db62fc6e46c474"} Jul 26 15:12:56.248895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:56.248762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" event={"ID":"c1dd4cbe-067a-491e-9d01-3636490460ff","Type":"ContainerStarted","Data":"4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1"} Jul 26 15:12:59.262888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:59.262849 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerID="01f3a3f1c87bf65f6700d505aafb29885acd7870194d7024f25bbab207619010" exitCode=0 Jul 26 15:12:59.263390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:12:59.262915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" event={"ID":"c1dd4cbe-067a-491e-9d01-3636490460ff","Type":"ContainerDied","Data":"01f3a3f1c87bf65f6700d505aafb29885acd7870194d7024f25bbab207619010"} Jul 26 15:13:00.268977 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:00.268942 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerID="c2e6967bd14a4a8ecb83ba7bdcf94e072005e73a7e6f97df02b0cbeaa2101e1a" exitCode=0 Jul 26 15:13:00.269398 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:00.269006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" event={"ID":"c1dd4cbe-067a-491e-9d01-3636490460ff","Type":"ContainerDied","Data":"c2e6967bd14a4a8ecb83ba7bdcf94e072005e73a7e6f97df02b0cbeaa2101e1a"} Jul 26 15:13:01.396041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.396012 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:13:01.470691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.470655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle\") pod \"c1dd4cbe-067a-491e-9d01-3636490460ff\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " Jul 26 15:13:01.470691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.470701 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc5gv\" (UniqueName: \"kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv\") pod \"c1dd4cbe-067a-491e-9d01-3636490460ff\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " Jul 26 15:13:01.470949 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.470730 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util\") pod \"c1dd4cbe-067a-491e-9d01-3636490460ff\" (UID: \"c1dd4cbe-067a-491e-9d01-3636490460ff\") " Jul 26 15:13:01.471179 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.471155 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle" (OuterVolumeSpecName: "bundle") pod "c1dd4cbe-067a-491e-9d01-3636490460ff" (UID: "c1dd4cbe-067a-491e-9d01-3636490460ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:13:01.472946 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.472926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv" (OuterVolumeSpecName: "kube-api-access-wc5gv") pod "c1dd4cbe-067a-491e-9d01-3636490460ff" (UID: "c1dd4cbe-067a-491e-9d01-3636490460ff"). InnerVolumeSpecName "kube-api-access-wc5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:13:01.475460 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.475409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util" (OuterVolumeSpecName: "util") pod "c1dd4cbe-067a-491e-9d01-3636490460ff" (UID: "c1dd4cbe-067a-491e-9d01-3636490460ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:13:01.572017 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.571930 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:01.572017 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.571956 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5gv\" (UniqueName: \"kubernetes.io/projected/c1dd4cbe-067a-491e-9d01-3636490460ff-kube-api-access-wc5gv\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:01.572017 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:01.571965 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1dd4cbe-067a-491e-9d01-3636490460ff-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:02.277783 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:02.277751 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" event={"ID":"c1dd4cbe-067a-491e-9d01-3636490460ff","Type":"ContainerDied","Data":"4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1"} Jul 26 15:13:02.277783 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:02.277773 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fngfkq" Jul 26 15:13:02.277994 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:02.277798 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5096ae72164431d7ae36f3070289b395e3c62eaf4b08527a3a975060c354a1" Jul 26 15:13:06.637897 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.637863 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-24sjf"] Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638154 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="pull" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638166 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="pull" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638177 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="extract" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638193 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="extract" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638205 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="util" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638213 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="util" Jul 26 15:13:06.638379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.638258 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dd4cbe-067a-491e-9d01-3636490460ff" containerName="extract" Jul 26 15:13:06.641268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.641251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.643940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.643915 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 26 15:13:06.644080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.643954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2qll6\"" Jul 26 15:13:06.645072 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.645055 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 26 15:13:06.649177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.649151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-24sjf"] Jul 26 15:13:06.710641 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.710609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-bound-sa-token\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.710837 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.710670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldvg\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-kube-api-access-vldvg\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.811287 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.811235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-bound-sa-token\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.811454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.811331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldvg\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-kube-api-access-vldvg\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.820457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.820421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-bound-sa-token\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.820714 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.820699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldvg\" (UniqueName: \"kubernetes.io/projected/6d77218f-ba81-4a31-87da-1f32802fe149-kube-api-access-vldvg\") pod \"cert-manager-55b4d954d5-24sjf\" (UID: \"6d77218f-ba81-4a31-87da-1f32802fe149\") " pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:06.951041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:06.951006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-24sjf" Jul 26 15:13:07.090955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:07.090920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-24sjf"] Jul 26 15:13:07.094560 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:07.094529 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d77218f_ba81_4a31_87da_1f32802fe149.slice/crio-999cbdc8f3000c42b908a1af8268497171e7225bd5d180bad38aa2ae63f51651 WatchSource:0}: Error finding container 999cbdc8f3000c42b908a1af8268497171e7225bd5d180bad38aa2ae63f51651: Status 404 returned error can't find the container with id 999cbdc8f3000c42b908a1af8268497171e7225bd5d180bad38aa2ae63f51651 Jul 26 15:13:07.295885 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:07.295802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-24sjf" event={"ID":"6d77218f-ba81-4a31-87da-1f32802fe149","Type":"ContainerStarted","Data":"999cbdc8f3000c42b908a1af8268497171e7225bd5d180bad38aa2ae63f51651"} Jul 26 15:13:10.314090 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:10.314047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-24sjf" event={"ID":"6d77218f-ba81-4a31-87da-1f32802fe149","Type":"ContainerStarted","Data":"0143bebd6e02477a7c3a6fc2f92aec626f22e1fa6de5e413394f97b5dec9f37c"} Jul 26 15:13:10.329802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:10.329751 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-24sjf" podStartSLOduration=1.707394673 podStartE2EDuration="4.329735546s" totalImagesPullingTime="2.622340873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:10.328805665 +0000 UTC m=+464.226480661" watchObservedRunningTime="2026-07-26 15:13:10.329735546 +0000 UTC m=+464.227410584" Jul 26 15:13:12.121681 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.121644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9"] Jul 26 15:13:12.124788 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.124772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.127383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.127356 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:13:12.128503 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.128480 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 15:13:12.128636 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.128522 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s24rq\"" Jul 26 15:13:12.132767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.132742 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9"] Jul 26 15:13:12.154053 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.154020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.154191 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.154066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5krc\" (UniqueName: \"kubernetes.io/projected/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-kube-api-access-z5krc\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.255066 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.255026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.255243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.255080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5krc\" (UniqueName: \"kubernetes.io/projected/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-kube-api-access-z5krc\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.255456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.255434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.264162 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.264133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5krc\" (UniqueName: \"kubernetes.io/projected/8d071fcf-7aa9-429b-b1b7-3a51ee3ec747-kube-api-access-z5krc\") pod \"openshift-lws-operator-fd8ccff4c-sq5x9\" (UID: \"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.434155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.434105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" Jul 26 15:13:12.578756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:12.578728 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9"] Jul 26 15:13:12.581336 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:12.581309 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d071fcf_7aa9_429b_b1b7_3a51ee3ec747.slice/crio-e042edf7d1eb8498182314995c4dcab1b752f6bf80ff2af913ea14458ed11824 WatchSource:0}: Error finding container e042edf7d1eb8498182314995c4dcab1b752f6bf80ff2af913ea14458ed11824: Status 404 returned error can't find the container with id e042edf7d1eb8498182314995c4dcab1b752f6bf80ff2af913ea14458ed11824 Jul 26 15:13:13.327778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:13.327740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" event={"ID":"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747","Type":"ContainerStarted","Data":"e042edf7d1eb8498182314995c4dcab1b752f6bf80ff2af913ea14458ed11824"} Jul 26 15:13:14.333300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:14.333246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" event={"ID":"8d071fcf-7aa9-429b-b1b7-3a51ee3ec747","Type":"ContainerStarted","Data":"068f49e1a61bb3040181c99f39ae2b99fb3786ffa413559cddbf43e89aab7b71"} Jul 26 15:13:14.350875 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:14.350814 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sq5x9" podStartSLOduration=0.669524487 podStartE2EDuration="2.350789428s" totalImagesPullingTime="1.681264941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:14.348770542 +0000 UTC m=+468.246445540" watchObservedRunningTime="2026-07-26 15:13:14.350789428 +0000 UTC m=+468.248464425" Jul 26 15:13:19.563232 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.563194 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7"] Jul 26 15:13:19.579886 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.579852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7"] Jul 26 15:13:19.580030 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.579993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.583014 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.582992 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5thlm\"" Jul 26 15:13:19.583160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.583031 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 26 15:13:19.583960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.583938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 26 15:13:19.610958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.610926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.611098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.610970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5xs\" (UniqueName: \"kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.611098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.610999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.712289 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.712237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.712456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.712301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5xs\" (UniqueName: \"kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.712456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.712337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.712659 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.712638 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.712708 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.712644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.720830 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.720803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5xs\" (UniqueName: \"kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:19.889367 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:19.889257 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:20.019255 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:20.019230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7"] Jul 26 15:13:20.021626 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:20.021601 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398c50c6_57bf_4c2f_952b_73837cb1f272.slice/crio-fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490 WatchSource:0}: Error finding container fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490: Status 404 returned error can't find the container with id fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490 Jul 26 15:13:20.356619 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:20.356582 2575 generic.go:356] "Generic (PLEG): container finished" podID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerID="ab7f3ddd7f4557d22f4a5a4591bbed7e006926811127a54a466c49f47f69e9cf" exitCode=0 Jul 26 15:13:20.356795 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:20.356617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" event={"ID":"398c50c6-57bf-4c2f-952b-73837cb1f272","Type":"ContainerDied","Data":"ab7f3ddd7f4557d22f4a5a4591bbed7e006926811127a54a466c49f47f69e9cf"} Jul 26 15:13:20.356795 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:20.356696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" event={"ID":"398c50c6-57bf-4c2f-952b-73837cb1f272","Type":"ContainerStarted","Data":"fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490"} Jul 26 15:13:21.362498 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:21.362404 2575 generic.go:356] "Generic (PLEG): container finished" podID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerID="45b183e210f41b3a3032d2466de6b21255b30e31ffdd646fc3e8f4521d3bc046" exitCode=0 Jul 26 15:13:21.362498 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:21.362464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" event={"ID":"398c50c6-57bf-4c2f-952b-73837cb1f272","Type":"ContainerDied","Data":"45b183e210f41b3a3032d2466de6b21255b30e31ffdd646fc3e8f4521d3bc046"} Jul 26 15:13:22.368574 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:22.368540 2575 generic.go:356] "Generic (PLEG): container finished" podID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerID="3910fc9a72078fb99894a375ea7f97c270f31be663c03e6ec7659cae27e863f1" exitCode=0 Jul 26 15:13:22.368954 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:22.368614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" event={"ID":"398c50c6-57bf-4c2f-952b-73837cb1f272","Type":"ContainerDied","Data":"3910fc9a72078fb99894a375ea7f97c270f31be663c03e6ec7659cae27e863f1"} Jul 26 15:13:23.494585 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.494560 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:23.547136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.547098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle\") pod \"398c50c6-57bf-4c2f-952b-73837cb1f272\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " Jul 26 15:13:23.547324 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.547185 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm5xs\" (UniqueName: \"kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs\") pod \"398c50c6-57bf-4c2f-952b-73837cb1f272\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " Jul 26 15:13:23.547324 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.547214 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util\") pod \"398c50c6-57bf-4c2f-952b-73837cb1f272\" (UID: \"398c50c6-57bf-4c2f-952b-73837cb1f272\") " Jul 26 15:13:23.547888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.547859 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle" (OuterVolumeSpecName: "bundle") pod "398c50c6-57bf-4c2f-952b-73837cb1f272" (UID: "398c50c6-57bf-4c2f-952b-73837cb1f272"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:13:23.549459 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.549429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs" (OuterVolumeSpecName: "kube-api-access-lm5xs") pod "398c50c6-57bf-4c2f-952b-73837cb1f272" (UID: "398c50c6-57bf-4c2f-952b-73837cb1f272"). InnerVolumeSpecName "kube-api-access-lm5xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:13:23.553028 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.553007 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util" (OuterVolumeSpecName: "util") pod "398c50c6-57bf-4c2f-952b-73837cb1f272" (UID: "398c50c6-57bf-4c2f-952b-73837cb1f272"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:13:23.648326 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.648202 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:23.648326 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.648236 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5xs\" (UniqueName: \"kubernetes.io/projected/398c50c6-57bf-4c2f-952b-73837cb1f272-kube-api-access-lm5xs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:23.648326 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:23.648249 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/398c50c6-57bf-4c2f-952b-73837cb1f272-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:13:24.378758 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:24.378727 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" Jul 26 15:13:24.378758 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:24.378750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aftbhl7" event={"ID":"398c50c6-57bf-4c2f-952b-73837cb1f272","Type":"ContainerDied","Data":"fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490"} Jul 26 15:13:24.379013 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:24.378804 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcc29b62de1da9a80160595c69d4f2d85c70d6fc240c02ee2ddbbf157022d490" Jul 26 15:13:32.484606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484571 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs"] Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484856 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="util" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484867 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="util" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484875 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="extract" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484880 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="extract" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484891 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="pull" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484897 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="pull" Jul 26 15:13:32.484998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.484950 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="398c50c6-57bf-4c2f-952b-73837cb1f272" containerName="extract" Jul 26 15:13:32.492119 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.492096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.495064 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.495032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 15:13:32.495064 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.495061 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 15:13:32.495344 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.495329 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7bw26\"" Jul 26 15:13:32.495415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.495378 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 15:13:32.495503 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.495486 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 15:13:32.508302 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.506935 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs"] Jul 26 15:13:32.623402 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.623366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.623574 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.623407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.623574 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.623472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.623648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.623581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.623648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.623635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9w7\" (UniqueName: \"kubernetes.io/projected/fa8cacf2-24fd-41eb-90a9-01590afc146c-kube-api-access-tf9w7\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9w7\" (UniqueName: \"kubernetes.io/projected/fa8cacf2-24fd-41eb-90a9-01590afc146c-kube-api-access-tf9w7\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.724642 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.724619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-manifests\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.726505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.726484 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa8cacf2-24fd-41eb-90a9-01590afc146c-tmp\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.726713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.726697 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.726779 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.726739 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa8cacf2-24fd-41eb-90a9-01590afc146c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.735780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.735722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9w7\" (UniqueName: \"kubernetes.io/projected/fa8cacf2-24fd-41eb-90a9-01590afc146c-kube-api-access-tf9w7\") pod \"opendatahub-operator-controller-manager-5c55b8954b-578rs\" (UID: \"fa8cacf2-24fd-41eb-90a9-01590afc146c\") " pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.804001 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.803965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:32.947131 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:32.947107 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs"] Jul 26 15:13:32.948632 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:32.948600 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8cacf2_24fd_41eb_90a9_01590afc146c.slice/crio-de98266e018273bb27af0b56f58e8444937ec5ef8e14ac46ec1cc641fe038aff WatchSource:0}: Error finding container de98266e018273bb27af0b56f58e8444937ec5ef8e14ac46ec1cc641fe038aff: Status 404 returned error can't find the container with id de98266e018273bb27af0b56f58e8444937ec5ef8e14ac46ec1cc641fe038aff Jul 26 15:13:33.414687 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:33.414648 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" event={"ID":"fa8cacf2-24fd-41eb-90a9-01590afc146c","Type":"ContainerStarted","Data":"de98266e018273bb27af0b56f58e8444937ec5ef8e14ac46ec1cc641fe038aff"} Jul 26 15:13:36.434995 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:36.434960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" event={"ID":"fa8cacf2-24fd-41eb-90a9-01590afc146c","Type":"ContainerStarted","Data":"c1053b677d70d9d446ea8f829c4ea26ccfecc902d169e081f4d7c1f97d6ed594"} Jul 26 15:13:37.440590 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:37.440488 2575 generic.go:356] "Generic (PLEG): container finished" podID="fa8cacf2-24fd-41eb-90a9-01590afc146c" containerID="c1053b677d70d9d446ea8f829c4ea26ccfecc902d169e081f4d7c1f97d6ed594" exitCode=0 Jul 26 15:13:37.440590 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:37.440584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" event={"ID":"fa8cacf2-24fd-41eb-90a9-01590afc146c","Type":"ContainerDied","Data":"c1053b677d70d9d446ea8f829c4ea26ccfecc902d169e081f4d7c1f97d6ed594"} Jul 26 15:13:38.446745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:38.446711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" event={"ID":"fa8cacf2-24fd-41eb-90a9-01590afc146c","Type":"ContainerStarted","Data":"1937277b8e789a0114a44ba6131f61727737f09b53a613d044fd5b72668ee9e5"} Jul 26 15:13:38.447144 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:38.446829 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:38.472364 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:38.472312 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" podStartSLOduration=1.923796244 podStartE2EDuration="6.472295634s" totalImagesPullingTime="3.537983336s" totalInitContainerRuntime="1.010516054s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:38.469266515 +0000 UTC m=+492.366941513" watchObservedRunningTime="2026-07-26 15:13:38.472295634 +0000 UTC m=+492.369970630" Jul 26 15:13:49.454498 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:49.454467 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c55b8954b-578rs" Jul 26 15:13:52.038150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.038101 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-tzznd"] Jul 26 15:13:52.042495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.042473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.046166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.046140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 15:13:52.046327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.046169 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 26 15:13:52.046327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.046174 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 15:13:52.047539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.047521 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 26 15:13:52.047738 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.047723 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9ldvq\"" Jul 26 15:13:52.060487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.060460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-tzznd"] Jul 26 15:13:52.190024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.189985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tls-certs\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.190249 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.190076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndz6d\" (UniqueName: \"kubernetes.io/projected/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-kube-api-access-ndz6d\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.190249 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.190115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tmp\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.291582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.291480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndz6d\" (UniqueName: \"kubernetes.io/projected/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-kube-api-access-ndz6d\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.291582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.291548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tmp\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.291811 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.291588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tls-certs\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.294082 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.294056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tmp\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.294334 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.294310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-tls-certs\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.309906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.309875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndz6d\" (UniqueName: \"kubernetes.io/projected/3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a-kube-api-access-ndz6d\") pod \"kube-auth-proxy-69f589b649-tzznd\" (UID: \"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a\") " pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.347829 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.347792 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdtfj"] Jul 26 15:13:52.351306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.351263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:52.351551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.351530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" Jul 26 15:13:52.354091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.354065 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 26 15:13:52.354091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.354075 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xdkcg\"" Jul 26 15:13:52.361560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.361533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdtfj"] Jul 26 15:13:52.493892 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.493852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vlp\" (UniqueName: \"kubernetes.io/projected/9376abe2-626d-44f6-8998-5e30259db5fc-kube-api-access-r4vlp\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:52.493892 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.493899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:52.498179 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.498152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69f589b649-tzznd"] Jul 26 15:13:52.500171 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:52.500142 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3326be6d_eb7f_4f1c_a65b_00d82fc4ef7a.slice/crio-47ea860dae51efa3dfba9d86683b756c2ddd871e982c65abe41b1e280866163b WatchSource:0}: Error finding container 47ea860dae51efa3dfba9d86683b756c2ddd871e982c65abe41b1e280866163b: Status 404 returned error can't find the container with id 47ea860dae51efa3dfba9d86683b756c2ddd871e982c65abe41b1e280866163b Jul 26 15:13:52.507494 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.507464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" event={"ID":"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a","Type":"ContainerStarted","Data":"47ea860dae51efa3dfba9d86683b756c2ddd871e982c65abe41b1e280866163b"} Jul 26 15:13:52.595259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.595160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vlp\" (UniqueName: \"kubernetes.io/projected/9376abe2-626d-44f6-8998-5e30259db5fc-kube-api-access-r4vlp\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:52.595259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.595204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:52.595508 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:13:52.595342 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 26 15:13:52.595508 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:13:52.595404 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert podName:9376abe2-626d-44f6-8998-5e30259db5fc nodeName:}" failed. No retries permitted until 2026-07-26 15:13:53.095387583 +0000 UTC m=+506.993062558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert") pod "odh-model-controller-7998c67dc4-fdtfj" (UID: "9376abe2-626d-44f6-8998-5e30259db5fc") : secret "odh-model-controller-webhook-cert" not found Jul 26 15:13:52.609955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:52.609920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vlp\" (UniqueName: \"kubernetes.io/projected/9376abe2-626d-44f6-8998-5e30259db5fc-kube-api-access-r4vlp\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:53.099299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.099249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:53.101846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.101822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9376abe2-626d-44f6-8998-5e30259db5fc-cert\") pod \"odh-model-controller-7998c67dc4-fdtfj\" (UID: \"9376abe2-626d-44f6-8998-5e30259db5fc\") " pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:53.263383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.263210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:53.278831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.278755 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:13:53.282630 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.282596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.286517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.286212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 26 15:13:53.286517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.286262 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 26 15:13:53.286517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.286212 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 15:13:53.286517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.286212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 26 15:13:53.286517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.286391 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2lvbp\"" Jul 26 15:13:53.299368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.299336 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:13:53.402259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwtx\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.402713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.402597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.426787 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.426696 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fdtfj"] Jul 26 15:13:53.430063 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:53.430030 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9376abe2_626d_44f6_8998_5e30259db5fc.slice/crio-48d51564198b6466afb771848e24854bfe2c2b20475ac450579c64f2c8c8ae7c WatchSource:0}: Error finding container 48d51564198b6466afb771848e24854bfe2c2b20475ac450579c64f2c8c8ae7c: Status 404 returned error can't find the container with id 48d51564198b6466afb771848e24854bfe2c2b20475ac450579c64f2c8c8ae7c Jul 26 15:13:53.503298 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwtx\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503619 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503619 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.503619 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.503570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.505062 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.505029 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.506556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.506529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.507968 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.507941 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.508317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.508240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.509814 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.509787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.512884 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.512852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwtx\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.522997 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.522969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6hrjm\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.524711 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.524680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" event={"ID":"9376abe2-626d-44f6-8998-5e30259db5fc","Type":"ContainerStarted","Data":"48d51564198b6466afb771848e24854bfe2c2b20475ac450579c64f2c8c8ae7c"} Jul 26 15:13:53.597912 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.597876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:53.776807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:53.776764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:13:53.779921 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:13:53.779886 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b01152_9303_4b29_82b5_81a59d76176d.slice/crio-994678bba47958c66f29b0019227e54df62c3d37550d9ec17dac72da3f97f9dc WatchSource:0}: Error finding container 994678bba47958c66f29b0019227e54df62c3d37550d9ec17dac72da3f97f9dc: Status 404 returned error can't find the container with id 994678bba47958c66f29b0019227e54df62c3d37550d9ec17dac72da3f97f9dc Jul 26 15:13:54.538492 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:54.538415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" event={"ID":"61b01152-9303-4b29-82b5-81a59d76176d","Type":"ContainerStarted","Data":"994678bba47958c66f29b0019227e54df62c3d37550d9ec17dac72da3f97f9dc"} Jul 26 15:13:58.143564 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.143528 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:13:58.143920 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.143608 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:13:58.559087 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.559050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" event={"ID":"3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a","Type":"ContainerStarted","Data":"5b679190ad389aa69a447651a18361f855e39b9c302911a727effad773d5dc75"} Jul 26 15:13:58.560603 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.560571 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" event={"ID":"61b01152-9303-4b29-82b5-81a59d76176d","Type":"ContainerStarted","Data":"0210d3c12063bf922e32722ee0ed5f5ee6d092dbf793e83ffc0787e4b7334324"} Jul 26 15:13:58.560740 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.560623 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:13:58.562038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.561966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" event={"ID":"9376abe2-626d-44f6-8998-5e30259db5fc","Type":"ContainerStarted","Data":"9186fafcaa14bf222b3689a70afa98357a5119599358050d24a063cb6b7c40a6"} Jul 26 15:13:58.562155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.562112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:13:58.575654 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.575598 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69f589b649-tzznd" podStartSLOduration=1.011068401 podStartE2EDuration="6.57557915s" totalImagesPullingTime="5.564510749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:58.574019702 +0000 UTC m=+512.471694700" watchObservedRunningTime="2026-07-26 15:13:58.57557915 +0000 UTC m=+512.473254154" Jul 26 15:13:58.591631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.591559 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" podStartSLOduration=1.942502464 podStartE2EDuration="6.591537045s" totalImagesPullingTime="4.649034581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:58.588897938 +0000 UTC m=+512.486572931" watchObservedRunningTime="2026-07-26 15:13:58.591537045 +0000 UTC m=+512.489212043" Jul 26 15:13:58.609549 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:58.609498 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" podStartSLOduration=1.248216719 podStartE2EDuration="5.60948077s" totalImagesPullingTime="4.361264051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:13:58.606513872 +0000 UTC m=+512.504188897" watchObservedRunningTime="2026-07-26 15:13:58.60948077 +0000 UTC m=+512.507155767" Jul 26 15:13:59.568376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:59.568339 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6hrjm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:13:59.568732 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:13:59.568386 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" podUID="61b01152-9303-4b29-82b5-81a59d76176d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:14:02.568322 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:02.568264 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:14:03.470200 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.470164 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-95c7j"] Jul 26 15:14:03.473743 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.473721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:03.477439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.477417 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 26 15:14:03.477566 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.477421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7lr54\"" Jul 26 15:14:03.478360 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.478339 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z"] Jul 26 15:14:03.481404 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.481382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.483898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.483879 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 26 15:14:03.484006 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.483915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7jkpj\"" Jul 26 15:14:03.486476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.486455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.486668 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.486647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6d4\" (UniqueName: \"kubernetes.io/projected/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-kube-api-access-mn6d4\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.486833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.486815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:03.486974 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.486955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zxw\" (UniqueName: \"kubernetes.io/projected/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-kube-api-access-z5zxw\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:03.489449 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.489195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-95c7j"] Jul 26 15:14:03.495471 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.495446 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z"] Jul 26 15:14:03.588054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.588021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.588062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6d4\" (UniqueName: \"kubernetes.io/projected/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-kube-api-access-mn6d4\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.588088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:03.588165 2575 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:03.588184 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.588205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zxw\" (UniqueName: \"kubernetes.io/projected/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-kube-api-access-z5zxw\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:03.588241 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert podName:f9ead486-31bc-4c2a-a673-e05b7ae1e5ff nodeName:}" failed. No retries permitted until 2026-07-26 15:14:04.088219915 +0000 UTC m=+517.985894907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert") pod "llmisvc-controller-manager-777c6bd7d5-lk99z" (UID: "f9ead486-31bc-4c2a-a673-e05b7ae1e5ff") : secret "llmisvc-webhook-server-cert" not found Jul 26 15:14:03.588549 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:03.588328 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert podName:c3c34933-b27a-4d7a-8c3c-d2c45d1476e7 nodeName:}" failed. No retries permitted until 2026-07-26 15:14:04.088303996 +0000 UTC m=+517.985978975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert") pod "kserve-controller-manager-d6f4b65b8-95c7j" (UID: "c3c34933-b27a-4d7a-8c3c-d2c45d1476e7") : secret "kserve-webhook-server-cert" not found Jul 26 15:14:03.599305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.599260 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6d4\" (UniqueName: \"kubernetes.io/projected/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-kube-api-access-mn6d4\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:03.599464 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:03.599386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zxw\" (UniqueName: \"kubernetes.io/projected/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-kube-api-access-z5zxw\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:04.093066 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.093026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:04.093250 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.093084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:04.095667 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.095644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9ead486-31bc-4c2a-a673-e05b7ae1e5ff-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lk99z\" (UID: \"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:04.095786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.095730 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c34933-b27a-4d7a-8c3c-d2c45d1476e7-cert\") pod \"kserve-controller-manager-d6f4b65b8-95c7j\" (UID: \"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:04.097524 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.097508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:04.241029 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.241005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z"] Jul 26 15:14:04.243206 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:14:04.243180 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9ead486_31bc_4c2a_a673_e05b7ae1e5ff.slice/crio-56a82f70e5fdf98aef89b2b9c1b5285cd741b63a436ee846c5df981a9a50d504 WatchSource:0}: Error finding container 56a82f70e5fdf98aef89b2b9c1b5285cd741b63a436ee846c5df981a9a50d504: Status 404 returned error can't find the container with id 56a82f70e5fdf98aef89b2b9c1b5285cd741b63a436ee846c5df981a9a50d504 Jul 26 15:14:04.383793 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.383700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:04.515969 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.515935 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-95c7j"] Jul 26 15:14:04.519610 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:14:04.519582 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c34933_b27a_4d7a_8c3c_d2c45d1476e7.slice/crio-c79a1d588641bc6c8c00c90897ddb72208640e1a463cd4398f828feb90de75eb WatchSource:0}: Error finding container c79a1d588641bc6c8c00c90897ddb72208640e1a463cd4398f828feb90de75eb: Status 404 returned error can't find the container with id c79a1d588641bc6c8c00c90897ddb72208640e1a463cd4398f828feb90de75eb Jul 26 15:14:04.591885 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.591844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" event={"ID":"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7","Type":"ContainerStarted","Data":"c79a1d588641bc6c8c00c90897ddb72208640e1a463cd4398f828feb90de75eb"} Jul 26 15:14:04.592982 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:04.592960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" event={"ID":"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff","Type":"ContainerStarted","Data":"56a82f70e5fdf98aef89b2b9c1b5285cd741b63a436ee846c5df981a9a50d504"} Jul 26 15:14:07.215396 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.215358 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:14:07.215867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.215601 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" podUID="61b01152-9303-4b29-82b5-81a59d76176d" containerName="discovery" containerID="cri-o://0210d3c12063bf922e32722ee0ed5f5ee6d092dbf793e83ffc0787e4b7334324" gracePeriod=30 Jul 26 15:14:07.602816 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:07.602761 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c34933_b27a_4d7a_8c3c_d2c45d1476e7.slice/crio-7d4aa1240d198e919520c2febf0c2f09001585433995f8316b65ebb145f6deba.scope\": RecentStats: unable to find data in memory cache]" Jul 26 15:14:07.612153 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.612120 2575 generic.go:356] "Generic (PLEG): container finished" podID="61b01152-9303-4b29-82b5-81a59d76176d" containerID="0210d3c12063bf922e32722ee0ed5f5ee6d092dbf793e83ffc0787e4b7334324" exitCode=0 Jul 26 15:14:07.612304 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.612248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" event={"ID":"61b01152-9303-4b29-82b5-81a59d76176d","Type":"ContainerDied","Data":"0210d3c12063bf922e32722ee0ed5f5ee6d092dbf793e83ffc0787e4b7334324"} Jul 26 15:14:07.664192 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.664167 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:14:07.733857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733822 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733870 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733889 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733952 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733981 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcwtx\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.733998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.734037 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts\") pod \"61b01152-9303-4b29-82b5-81a59d76176d\" (UID: \"61b01152-9303-4b29-82b5-81a59d76176d\") " Jul 26 15:14:07.734421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.734368 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:14:07.736696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:14:07.736899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736717 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:14:07.736899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736728 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token" (OuterVolumeSpecName: "istio-token") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:14:07.736899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736775 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx" (OuterVolumeSpecName: "kube-api-access-jcwtx") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "kube-api-access-jcwtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:14:07.736899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts" (OuterVolumeSpecName: "cacerts") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:14:07.736899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.736804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs" (OuterVolumeSpecName: "local-certs") pod "61b01152-9303-4b29-82b5-81a59d76176d" (UID: "61b01152-9303-4b29-82b5-81a59d76176d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835107 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-dns-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835135 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcwtx\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-kube-api-access-jcwtx\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835144 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-istio-kubeconfig\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835153 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/61b01152-9303-4b29-82b5-81a59d76176d-cacerts\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835161 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b01152-9303-4b29-82b5-81a59d76176d-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835168 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b01152-9303-4b29-82b5-81a59d76176d-local-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:07.835187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:07.835177 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/61b01152-9303-4b29-82b5-81a59d76176d-istio-csr-ca-configmap\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:08.618975 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.618929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" event={"ID":"c3c34933-b27a-4d7a-8c3c-d2c45d1476e7","Type":"ContainerStarted","Data":"7d4aa1240d198e919520c2febf0c2f09001585433995f8316b65ebb145f6deba"} Jul 26 15:14:08.619444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.619039 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:08.620457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.620431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" event={"ID":"61b01152-9303-4b29-82b5-81a59d76176d","Type":"ContainerDied","Data":"994678bba47958c66f29b0019227e54df62c3d37550d9ec17dac72da3f97f9dc"} Jul 26 15:14:08.620599 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.620436 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm" Jul 26 15:14:08.620599 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.620483 2575 scope.go:117] "RemoveContainer" containerID="0210d3c12063bf922e32722ee0ed5f5ee6d092dbf793e83ffc0787e4b7334324" Jul 26 15:14:08.622137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.622111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" event={"ID":"f9ead486-31bc-4c2a-a673-e05b7ae1e5ff","Type":"ContainerStarted","Data":"d6529091619738f6458c12bb1e74315964d6e6b6f196a5eab80d71799a6345d9"} Jul 26 15:14:08.622269 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.622250 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:08.635820 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.635777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" podStartSLOduration=2.622907588 podStartE2EDuration="5.635764073s" totalImagesPullingTime="3.012856485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:14:08.634964319 +0000 UTC m=+522.532639317" watchObservedRunningTime="2026-07-26 15:14:08.635764073 +0000 UTC m=+522.533439069" Jul 26 15:14:08.666292 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.666223 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" podStartSLOduration=2.372308308 podStartE2EDuration="5.666209278s" totalImagesPullingTime="3.29390097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:14:08.6654177 +0000 UTC m=+522.563092696" watchObservedRunningTime="2026-07-26 15:14:08.666209278 +0000 UTC m=+522.563884274" Jul 26 15:14:08.699129 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.699092 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:14:08.712720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:08.712690 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6hrjm"] Jul 26 15:14:09.569663 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:09.569635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fdtfj" Jul 26 15:14:10.553821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:10.553791 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b01152-9303-4b29-82b5-81a59d76176d" path="/var/lib/kubelet/pods/61b01152-9303-4b29-82b5-81a59d76176d/volumes" Jul 26 15:14:39.629808 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:39.629779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lk99z" Jul 26 15:14:39.632955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:39.632934 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-95c7j" Jul 26 15:14:53.173839 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.173804 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:53.174251 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.174205 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b01152-9303-4b29-82b5-81a59d76176d" containerName="discovery" Jul 26 15:14:53.174251 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.174219 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b01152-9303-4b29-82b5-81a59d76176d" containerName="discovery" Jul 26 15:14:53.174373 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.174303 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b01152-9303-4b29-82b5-81a59d76176d" containerName="discovery" Jul 26 15:14:53.177243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.177218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:53.179777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.179752 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 15:14:53.179923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.179822 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 15:14:53.181050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.181032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-w8jkr\"" Jul 26 15:14:53.188709 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.188686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:53.209120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.209085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr64l\" (UniqueName: \"kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l\") pod \"kuadrant-operator-catalog-cdwj8\" (UID: \"a09bcf40-bb20-49cd-b521-73af18390f3b\") " pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:53.310009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.309970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr64l\" (UniqueName: \"kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l\") pod \"kuadrant-operator-catalog-cdwj8\" (UID: \"a09bcf40-bb20-49cd-b521-73af18390f3b\") " pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:53.319606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.319573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr64l\" (UniqueName: \"kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l\") pod \"kuadrant-operator-catalog-cdwj8\" (UID: \"a09bcf40-bb20-49cd-b521-73af18390f3b\") " pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:53.488191 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.488154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:53.533187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.533151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:53.622462 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.622438 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:53.624846 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:14:53.624820 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09bcf40_bb20_49cd_b521_73af18390f3b.slice/crio-aef39acca52dbf47e2b3ac70c97a878f799d07b6642073761f33a136b3a74c2f WatchSource:0}: Error finding container aef39acca52dbf47e2b3ac70c97a878f799d07b6642073761f33a136b3a74c2f: Status 404 returned error can't find the container with id aef39acca52dbf47e2b3ac70c97a878f799d07b6642073761f33a136b3a74c2f Jul 26 15:14:53.746513 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.746431 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-45ccx"] Jul 26 15:14:53.750646 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.750628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:14:53.757511 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.757468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-45ccx"] Jul 26 15:14:53.814188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.814156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjstj\" (UniqueName: \"kubernetes.io/projected/96472d48-c6f8-4c04-9489-aa7ff496c1e7-kube-api-access-zjstj\") pod \"kuadrant-operator-catalog-45ccx\" (UID: \"96472d48-c6f8-4c04-9489-aa7ff496c1e7\") " pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:14:53.820416 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.820378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" event={"ID":"a09bcf40-bb20-49cd-b521-73af18390f3b","Type":"ContainerStarted","Data":"aef39acca52dbf47e2b3ac70c97a878f799d07b6642073761f33a136b3a74c2f"} Jul 26 15:14:53.915558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.915521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjstj\" (UniqueName: \"kubernetes.io/projected/96472d48-c6f8-4c04-9489-aa7ff496c1e7-kube-api-access-zjstj\") pod \"kuadrant-operator-catalog-45ccx\" (UID: \"96472d48-c6f8-4c04-9489-aa7ff496c1e7\") " pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:14:53.923351 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:53.923321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjstj\" (UniqueName: \"kubernetes.io/projected/96472d48-c6f8-4c04-9489-aa7ff496c1e7-kube-api-access-zjstj\") pod \"kuadrant-operator-catalog-45ccx\" (UID: \"96472d48-c6f8-4c04-9489-aa7ff496c1e7\") " pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:14:54.061604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:54.061522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:14:54.193556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:54.193519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-45ccx"] Jul 26 15:14:54.196854 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:14:54.196826 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96472d48_c6f8_4c04_9489_aa7ff496c1e7.slice/crio-d0dbc117910655db218bc30be203236391fc0bd08cae38c92b0e96da14d8d46b WatchSource:0}: Error finding container d0dbc117910655db218bc30be203236391fc0bd08cae38c92b0e96da14d8d46b: Status 404 returned error can't find the container with id d0dbc117910655db218bc30be203236391fc0bd08cae38c92b0e96da14d8d46b Jul 26 15:14:54.828657 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:54.828613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" event={"ID":"96472d48-c6f8-4c04-9489-aa7ff496c1e7","Type":"ContainerStarted","Data":"d0dbc117910655db218bc30be203236391fc0bd08cae38c92b0e96da14d8d46b"} Jul 26 15:14:56.839495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:56.839460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" event={"ID":"a09bcf40-bb20-49cd-b521-73af18390f3b","Type":"ContainerStarted","Data":"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41"} Jul 26 15:14:56.839961 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:56.839488 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" podUID="a09bcf40-bb20-49cd-b521-73af18390f3b" containerName="registry-server" containerID="cri-o://c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41" gracePeriod=2 Jul 26 15:14:56.841048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:56.841025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" event={"ID":"96472d48-c6f8-4c04-9489-aa7ff496c1e7","Type":"ContainerStarted","Data":"25903392a5f3edb1983e1333d73bc0f3457dceac750e1a1837e0d3f8d82cd0d4"} Jul 26 15:14:56.856267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:56.856217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" podStartSLOduration=1.713938441 podStartE2EDuration="3.856201714s" totalImagesPullingTime="2.142263273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:14:56.855563027 +0000 UTC m=+570.753238025" watchObservedRunningTime="2026-07-26 15:14:56.856201714 +0000 UTC m=+570.753876711" Jul 26 15:14:56.870818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:56.870768 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" podStartSLOduration=2.297203564 podStartE2EDuration="3.87075328s" totalImagesPullingTime="1.573549716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:14:56.869243014 +0000 UTC m=+570.766918023" watchObservedRunningTime="2026-07-26 15:14:56.87075328 +0000 UTC m=+570.768428276" Jul 26 15:14:57.082220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.082193 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:57.148400 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.148304 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr64l\" (UniqueName: \"kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l\") pod \"a09bcf40-bb20-49cd-b521-73af18390f3b\" (UID: \"a09bcf40-bb20-49cd-b521-73af18390f3b\") " Jul 26 15:14:57.150636 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.150612 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l" (OuterVolumeSpecName: "kube-api-access-tr64l") pod "a09bcf40-bb20-49cd-b521-73af18390f3b" (UID: "a09bcf40-bb20-49cd-b521-73af18390f3b"). InnerVolumeSpecName "kube-api-access-tr64l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:14:57.249844 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.249804 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr64l\" (UniqueName: \"kubernetes.io/projected/a09bcf40-bb20-49cd-b521-73af18390f3b-kube-api-access-tr64l\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:14:57.849568 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.849530 2575 generic.go:356] "Generic (PLEG): container finished" podID="a09bcf40-bb20-49cd-b521-73af18390f3b" containerID="c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41" exitCode=0 Jul 26 15:14:57.850061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.849575 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" Jul 26 15:14:57.850061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.849606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" event={"ID":"a09bcf40-bb20-49cd-b521-73af18390f3b","Type":"ContainerDied","Data":"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41"} Jul 26 15:14:57.850061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.849658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cdwj8" event={"ID":"a09bcf40-bb20-49cd-b521-73af18390f3b","Type":"ContainerDied","Data":"aef39acca52dbf47e2b3ac70c97a878f799d07b6642073761f33a136b3a74c2f"} Jul 26 15:14:57.850061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.849678 2575 scope.go:117] "RemoveContainer" containerID="c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41" Jul 26 15:14:57.858328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.858313 2575 scope.go:117] "RemoveContainer" containerID="c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41" Jul 26 15:14:57.858561 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:14:57.858540 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41\": container with ID starting with c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41 not found: ID does not exist" containerID="c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41" Jul 26 15:14:57.858635 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.858572 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41"} err="failed to get container status \"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41\": rpc error: code = NotFound desc = could not find container \"c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41\": container with ID starting with c34dcb13dc923f91cd62c3c60d45241f59c13391127fae1b0433ba0a81171d41 not found: ID does not exist" Jul 26 15:14:57.870431 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.870398 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:57.872285 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:57.872250 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cdwj8"] Jul 26 15:14:58.554096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:14:58.554051 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09bcf40-bb20-49cd-b521-73af18390f3b" path="/var/lib/kubelet/pods/a09bcf40-bb20-49cd-b521-73af18390f3b/volumes" Jul 26 15:15:04.062401 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:04.062362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:15:04.062815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:04.062416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:15:04.084079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:04.084053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:15:04.902379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:04.902265 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-45ccx" Jul 26 15:15:08.351005 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.350967 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr"] Jul 26 15:15:08.351408 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.351288 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09bcf40-bb20-49cd-b521-73af18390f3b" containerName="registry-server" Jul 26 15:15:08.351408 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.351303 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09bcf40-bb20-49cd-b521-73af18390f3b" containerName="registry-server" Jul 26 15:15:08.351408 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.351369 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09bcf40-bb20-49cd-b521-73af18390f3b" containerName="registry-server" Jul 26 15:15:08.355844 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.355824 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.358263 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.358241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6r862\"" Jul 26 15:15:08.361431 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.361409 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr"] Jul 26 15:15:08.446362 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.446324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.446362 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.446367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zj9\" (UniqueName: \"kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.446578 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.446433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.547421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.547383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.547606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.547445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.547606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.547474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zj9\" (UniqueName: \"kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.547791 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.547749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.547879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.547829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.555818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.555788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zj9\" (UniqueName: \"kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.666721 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.666635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:08.802318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.802291 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr"] Jul 26 15:15:08.803951 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:08.803925 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e0a00d_05c2_4331_a69b_3b70ffe86777.slice/crio-06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536 WatchSource:0}: Error finding container 06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536: Status 404 returned error can't find the container with id 06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536 Jul 26 15:15:08.899620 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.899578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerStarted","Data":"ccbe50997ab3a0a54d269d0cb71d39bc518122cf6abf068a6de7421a8528b267"} Jul 26 15:15:08.899774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:08.899650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerStarted","Data":"06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536"} Jul 26 15:15:09.146867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.146827 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh"] Jul 26 15:15:09.150247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.150224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.158301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.158254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh"] Jul 26 15:15:09.252694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.252658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.252694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.252699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng54\" (UniqueName: \"kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.252908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.252732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.353570 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.353531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.353570 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.353572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng54\" (UniqueName: \"kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.354024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.353708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.354024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.353963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.354024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.354013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.365440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.365418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng54\" (UniqueName: \"kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.460219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.460187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:09.549896 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.549862 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9"] Jul 26 15:15:09.555031 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.555005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.571501 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.570864 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9"] Jul 26 15:15:09.602810 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.602687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh"] Jul 26 15:15:09.605448 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:09.605417 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615247b9_2b38_427b_bcbd_5c21ae1388ce.slice/crio-e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410 WatchSource:0}: Error finding container e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410: Status 404 returned error can't find the container with id e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410 Jul 26 15:15:09.656315 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.656258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.656436 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.656383 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.656508 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.656487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8d6\" (UniqueName: \"kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.757053 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.756969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.757053 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.757026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8d6\" (UniqueName: \"kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.757231 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.757056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.757440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.757420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.757499 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.757423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.765250 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.765225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8d6\" (UniqueName: \"kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.884193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.884151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:09.905766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.905725 2575 generic.go:356] "Generic (PLEG): container finished" podID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerID="86f52317423f814bb0f9f105b31576acf9f20a2db8bb3dc5ae715f1d3c8c916d" exitCode=0 Jul 26 15:15:09.905923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.905861 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" event={"ID":"615247b9-2b38-427b-bcbd-5c21ae1388ce","Type":"ContainerDied","Data":"86f52317423f814bb0f9f105b31576acf9f20a2db8bb3dc5ae715f1d3c8c916d"} Jul 26 15:15:09.905923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.905911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" event={"ID":"615247b9-2b38-427b-bcbd-5c21ae1388ce","Type":"ContainerStarted","Data":"e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410"} Jul 26 15:15:09.907961 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.907926 2575 generic.go:356] "Generic (PLEG): container finished" podID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerID="ccbe50997ab3a0a54d269d0cb71d39bc518122cf6abf068a6de7421a8528b267" exitCode=0 Jul 26 15:15:09.908077 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.908024 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerDied","Data":"ccbe50997ab3a0a54d269d0cb71d39bc518122cf6abf068a6de7421a8528b267"} Jul 26 15:15:09.953313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.953249 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs"] Jul 26 15:15:09.957713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.957692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:09.964536 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:09.964509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs"] Jul 26 15:15:10.027182 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.027159 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9"] Jul 26 15:15:10.028763 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:10.028742 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dccb90a_9e1e_4178_8781_5e0df1404e00.slice/crio-a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a WatchSource:0}: Error finding container a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a: Status 404 returned error can't find the container with id a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a Jul 26 15:15:10.059772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.059738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqlhg\" (UniqueName: \"kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.059891 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.059842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.059891 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.059879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.160828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.160790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.160993 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.160838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.160993 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.160869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqlhg\" (UniqueName: \"kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.161214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.161192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.161252 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.161202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.169906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.169879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqlhg\" (UniqueName: \"kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.269036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.269003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:10.401821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.401798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs"] Jul 26 15:15:10.403558 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:10.403531 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f79418b_8fd7_4226_b8a8_0c5472de6cdb.slice/crio-51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0 WatchSource:0}: Error finding container 51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0: Status 404 returned error can't find the container with id 51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0 Jul 26 15:15:10.917454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.917417 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerID="3c9c64609e20f6fc9315328bc2a16ac94230a12e1f994491192872ce07995bcb" exitCode=0 Jul 26 15:15:10.917648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.917503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" event={"ID":"5f79418b-8fd7-4226-b8a8-0c5472de6cdb","Type":"ContainerDied","Data":"3c9c64609e20f6fc9315328bc2a16ac94230a12e1f994491192872ce07995bcb"} Jul 26 15:15:10.917648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.917554 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" event={"ID":"5f79418b-8fd7-4226-b8a8-0c5472de6cdb","Type":"ContainerStarted","Data":"51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0"} Jul 26 15:15:10.919039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.919011 2575 generic.go:356] "Generic (PLEG): container finished" podID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerID="b24c5f9dd33ee54a7be3f53b85208aa0ceb867588dae1411dc57cbc9680db8ee" exitCode=0 Jul 26 15:15:10.919135 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.919059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" event={"ID":"0dccb90a-9e1e-4178-8781-5e0df1404e00","Type":"ContainerDied","Data":"b24c5f9dd33ee54a7be3f53b85208aa0ceb867588dae1411dc57cbc9680db8ee"} Jul 26 15:15:10.919135 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.919125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" event={"ID":"0dccb90a-9e1e-4178-8781-5e0df1404e00","Type":"ContainerStarted","Data":"a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a"} Jul 26 15:15:10.921236 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.921140 2575 generic.go:356] "Generic (PLEG): container finished" podID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerID="30a4d35683a2ab806cae7aab74074dfe888d8d92440c4ece576432922110d460" exitCode=0 Jul 26 15:15:10.921327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.921237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" event={"ID":"615247b9-2b38-427b-bcbd-5c21ae1388ce","Type":"ContainerDied","Data":"30a4d35683a2ab806cae7aab74074dfe888d8d92440c4ece576432922110d460"} Jul 26 15:15:10.923244 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.923221 2575 generic.go:356] "Generic (PLEG): container finished" podID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerID="71125f6b0326a9c6b9c67bd132dc7788ac8031a9f963cda25a3ac96625beae48" exitCode=0 Jul 26 15:15:10.923424 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:10.923305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerDied","Data":"71125f6b0326a9c6b9c67bd132dc7788ac8031a9f963cda25a3ac96625beae48"} Jul 26 15:15:11.931235 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.931202 2575 generic.go:356] "Generic (PLEG): container finished" podID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerID="28689cd41b8dd77410fc084f134e02e0da05af723862c4d8c94201f2fb498d42" exitCode=0 Jul 26 15:15:11.931667 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.931322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerDied","Data":"28689cd41b8dd77410fc084f134e02e0da05af723862c4d8c94201f2fb498d42"} Jul 26 15:15:11.933012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.932990 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerID="589183e1be4a4fe824fdd382665668ce3b564a7b512cf9a26b4f7020323ef815" exitCode=0 Jul 26 15:15:11.933147 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.933017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" event={"ID":"5f79418b-8fd7-4226-b8a8-0c5472de6cdb","Type":"ContainerDied","Data":"589183e1be4a4fe824fdd382665668ce3b564a7b512cf9a26b4f7020323ef815"} Jul 26 15:15:11.935056 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.935031 2575 generic.go:356] "Generic (PLEG): container finished" podID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerID="167e7a8820e28d96c9f4f8e45f8d76029d04e77176e8f445a2e7b3cd541ef73b" exitCode=0 Jul 26 15:15:11.935152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.935114 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" event={"ID":"0dccb90a-9e1e-4178-8781-5e0df1404e00","Type":"ContainerDied","Data":"167e7a8820e28d96c9f4f8e45f8d76029d04e77176e8f445a2e7b3cd541ef73b"} Jul 26 15:15:11.937300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.937265 2575 generic.go:356] "Generic (PLEG): container finished" podID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerID="570b10eb848080a1712e9c854030e4f9a5bbb5c6587a70777274d5ecd4cd31c1" exitCode=0 Jul 26 15:15:11.937388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:11.937313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" event={"ID":"615247b9-2b38-427b-bcbd-5c21ae1388ce","Type":"ContainerDied","Data":"570b10eb848080a1712e9c854030e4f9a5bbb5c6587a70777274d5ecd4cd31c1"} Jul 26 15:15:12.947174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:12.947142 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerID="7a1e9cd7e9642f0fabff93ae8d98ffe2ab0bab3be493b3f8fbececbb8973d829" exitCode=0 Jul 26 15:15:12.947629 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:12.947220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" event={"ID":"5f79418b-8fd7-4226-b8a8-0c5472de6cdb","Type":"ContainerDied","Data":"7a1e9cd7e9642f0fabff93ae8d98ffe2ab0bab3be493b3f8fbececbb8973d829"} Jul 26 15:15:12.949220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:12.949195 2575 generic.go:356] "Generic (PLEG): container finished" podID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerID="fa98a9beb4cc6171e1dab2d00f54b620c40f88a999635dd4dadf98e5bb239cde" exitCode=0 Jul 26 15:15:12.949340 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:12.949291 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" event={"ID":"0dccb90a-9e1e-4178-8781-5e0df1404e00","Type":"ContainerDied","Data":"fa98a9beb4cc6171e1dab2d00f54b620c40f88a999635dd4dadf98e5bb239cde"} Jul 26 15:15:13.089222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.089196 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:13.122501 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.122472 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:13.186108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8zj9\" (UniqueName: \"kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9\") pod \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " Jul 26 15:15:13.186108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186107 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle\") pod \"615247b9-2b38-427b-bcbd-5c21ae1388ce\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " Jul 26 15:15:13.186356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186131 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle\") pod \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " Jul 26 15:15:13.186356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wng54\" (UniqueName: \"kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54\") pod \"615247b9-2b38-427b-bcbd-5c21ae1388ce\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " Jul 26 15:15:13.186356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util\") pod \"615247b9-2b38-427b-bcbd-5c21ae1388ce\" (UID: \"615247b9-2b38-427b-bcbd-5c21ae1388ce\") " Jul 26 15:15:13.186356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186318 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util\") pod \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\" (UID: \"f9e0a00d-05c2-4331-a69b-3b70ffe86777\") " Jul 26 15:15:13.186906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186847 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle" (OuterVolumeSpecName: "bundle") pod "615247b9-2b38-427b-bcbd-5c21ae1388ce" (UID: "615247b9-2b38-427b-bcbd-5c21ae1388ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:13.187005 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.186926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle" (OuterVolumeSpecName: "bundle") pod "f9e0a00d-05c2-4331-a69b-3b70ffe86777" (UID: "f9e0a00d-05c2-4331-a69b-3b70ffe86777"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:13.188490 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.188467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9" (OuterVolumeSpecName: "kube-api-access-m8zj9") pod "f9e0a00d-05c2-4331-a69b-3b70ffe86777" (UID: "f9e0a00d-05c2-4331-a69b-3b70ffe86777"). InnerVolumeSpecName "kube-api-access-m8zj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:13.188570 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.188541 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54" (OuterVolumeSpecName: "kube-api-access-wng54") pod "615247b9-2b38-427b-bcbd-5c21ae1388ce" (UID: "615247b9-2b38-427b-bcbd-5c21ae1388ce"). InnerVolumeSpecName "kube-api-access-wng54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:13.194897 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.194871 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util" (OuterVolumeSpecName: "util") pod "615247b9-2b38-427b-bcbd-5c21ae1388ce" (UID: "615247b9-2b38-427b-bcbd-5c21ae1388ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:13.195152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.195135 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util" (OuterVolumeSpecName: "util") pod "f9e0a00d-05c2-4331-a69b-3b70ffe86777" (UID: "f9e0a00d-05c2-4331-a69b-3b70ffe86777"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287816 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zj9\" (UniqueName: \"kubernetes.io/projected/f9e0a00d-05c2-4331-a69b-3b70ffe86777-kube-api-access-m8zj9\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287848 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287858 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287865 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wng54\" (UniqueName: \"kubernetes.io/projected/615247b9-2b38-427b-bcbd-5c21ae1388ce-kube-api-access-wng54\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287874 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/615247b9-2b38-427b-bcbd-5c21ae1388ce-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.287906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.287883 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9e0a00d-05c2-4331-a69b-3b70ffe86777-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:13.955380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.955348 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" Jul 26 15:15:13.955380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.955360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr" event={"ID":"f9e0a00d-05c2-4331-a69b-3b70ffe86777","Type":"ContainerDied","Data":"06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536"} Jul 26 15:15:13.955901 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.955402 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a77510f39989aaaa5db758bb3768130584d7725d7942f745f5a919f0800536" Jul 26 15:15:13.957033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.957014 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" Jul 26 15:15:13.957167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.957047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh" event={"ID":"615247b9-2b38-427b-bcbd-5c21ae1388ce","Type":"ContainerDied","Data":"e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410"} Jul 26 15:15:13.957167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:13.957080 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e37e717d4ae04af35f694aa5300d0b57a4e2cb8ebe7c93bf6019d228372410" Jul 26 15:15:14.095675 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.095649 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:14.116915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.116881 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:14.196553 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196519 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle\") pod \"0dccb90a-9e1e-4178-8781-5e0df1404e00\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " Jul 26 15:15:14.196553 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util\") pod \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " Jul 26 15:15:14.196770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196586 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle\") pod \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " Jul 26 15:15:14.196770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196658 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util\") pod \"0dccb90a-9e1e-4178-8781-5e0df1404e00\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " Jul 26 15:15:14.196770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196689 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqlhg\" (UniqueName: \"kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg\") pod \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\" (UID: \"5f79418b-8fd7-4226-b8a8-0c5472de6cdb\") " Jul 26 15:15:14.196770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.196721 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm8d6\" (UniqueName: \"kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6\") pod \"0dccb90a-9e1e-4178-8781-5e0df1404e00\" (UID: \"0dccb90a-9e1e-4178-8781-5e0df1404e00\") " Jul 26 15:15:14.197177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.197145 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle" (OuterVolumeSpecName: "bundle") pod "0dccb90a-9e1e-4178-8781-5e0df1404e00" (UID: "0dccb90a-9e1e-4178-8781-5e0df1404e00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:14.197177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.197163 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle" (OuterVolumeSpecName: "bundle") pod "5f79418b-8fd7-4226-b8a8-0c5472de6cdb" (UID: "5f79418b-8fd7-4226-b8a8-0c5472de6cdb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:14.199150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.199125 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6" (OuterVolumeSpecName: "kube-api-access-wm8d6") pod "0dccb90a-9e1e-4178-8781-5e0df1404e00" (UID: "0dccb90a-9e1e-4178-8781-5e0df1404e00"). InnerVolumeSpecName "kube-api-access-wm8d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:14.199245 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.199146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg" (OuterVolumeSpecName: "kube-api-access-qqlhg") pod "5f79418b-8fd7-4226-b8a8-0c5472de6cdb" (UID: "5f79418b-8fd7-4226-b8a8-0c5472de6cdb"). InnerVolumeSpecName "kube-api-access-qqlhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:14.203070 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.203047 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util" (OuterVolumeSpecName: "util") pod "5f79418b-8fd7-4226-b8a8-0c5472de6cdb" (UID: "5f79418b-8fd7-4226-b8a8-0c5472de6cdb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:14.203070 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.203059 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util" (OuterVolumeSpecName: "util") pod "0dccb90a-9e1e-4178-8781-5e0df1404e00" (UID: "0dccb90a-9e1e-4178-8781-5e0df1404e00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298264 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqlhg\" (UniqueName: \"kubernetes.io/projected/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-kube-api-access-qqlhg\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298327 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8d6\" (UniqueName: \"kubernetes.io/projected/0dccb90a-9e1e-4178-8781-5e0df1404e00-kube-api-access-wm8d6\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298337 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298345 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298353 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f79418b-8fd7-4226-b8a8-0c5472de6cdb-bundle\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.298386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.298360 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dccb90a-9e1e-4178-8781-5e0df1404e00-util\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:14.964049 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.964021 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" Jul 26 15:15:14.964049 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.964019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs" event={"ID":"5f79418b-8fd7-4226-b8a8-0c5472de6cdb","Type":"ContainerDied","Data":"51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0"} Jul 26 15:15:14.964529 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.964082 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51f2e91f2677caf9127a726f2fd45aad28971a9ba32363630939026d4708c8b0" Jul 26 15:15:14.968883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.968844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" event={"ID":"0dccb90a-9e1e-4178-8781-5e0df1404e00","Type":"ContainerDied","Data":"a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a"} Jul 26 15:15:14.969015 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.968927 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5cf0c10b796acbc1083dab84c6f3fa146ad3d3b8b25a119a4ccd7909f0b2c5a" Jul 26 15:15:14.969090 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:14.969031 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9" Jul 26 15:15:19.679487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679443 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh"] Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679884 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="util" Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679903 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="util" Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679922 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="pull" Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679930 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="pull" Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679939 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="util" Jul 26 15:15:19.679951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679947 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="util" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679958 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679968 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679980 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="util" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.679988 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="util" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680000 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680008 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680024 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680042 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="util" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680050 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="util" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680060 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680067 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680078 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680085 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="pull" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680099 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680107 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680118 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680125 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680203 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="615247b9-2b38-427b-bcbd-5c21ae1388ce" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680217 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e0a00d-05c2-4331-a69b-3b70ffe86777" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680226 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f79418b-8fd7-4226-b8a8-0c5472de6cdb" containerName="extract" Jul 26 15:15:19.680264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.680238 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dccb90a-9e1e-4178-8781-5e0df1404e00" containerName="extract" Jul 26 15:15:19.683353 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.683332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:19.685940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.685918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pwpw7\"" Jul 26 15:15:19.692978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.692945 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh"] Jul 26 15:15:19.740201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.740161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlmbs\" (UniqueName: \"kubernetes.io/projected/3fb0e491-250d-4840-8da6-84203acbe577-kube-api-access-jlmbs\") pod \"authorino-operator-75cc8d5cb7-wh8hh\" (UID: \"3fb0e491-250d-4840-8da6-84203acbe577\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:19.840688 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.840646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlmbs\" (UniqueName: \"kubernetes.io/projected/3fb0e491-250d-4840-8da6-84203acbe577-kube-api-access-jlmbs\") pod \"authorino-operator-75cc8d5cb7-wh8hh\" (UID: \"3fb0e491-250d-4840-8da6-84203acbe577\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:19.849388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.849355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlmbs\" (UniqueName: \"kubernetes.io/projected/3fb0e491-250d-4840-8da6-84203acbe577-kube-api-access-jlmbs\") pod \"authorino-operator-75cc8d5cb7-wh8hh\" (UID: \"3fb0e491-250d-4840-8da6-84203acbe577\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:19.994522 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:19.994435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:20.128317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:20.128263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh"] Jul 26 15:15:20.130477 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:20.130440 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb0e491_250d_4840_8da6_84203acbe577.slice/crio-85ea3be40184ae51679a996a5932186709d4b9b0e5f1f767349ca2ce0bb5829b WatchSource:0}: Error finding container 85ea3be40184ae51679a996a5932186709d4b9b0e5f1f767349ca2ce0bb5829b: Status 404 returned error can't find the container with id 85ea3be40184ae51679a996a5932186709d4b9b0e5f1f767349ca2ce0bb5829b Jul 26 15:15:20.997593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:20.997548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" event={"ID":"3fb0e491-250d-4840-8da6-84203acbe577","Type":"ContainerStarted","Data":"85ea3be40184ae51679a996a5932186709d4b9b0e5f1f767349ca2ce0bb5829b"} Jul 26 15:15:22.525736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.525684 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm"] Jul 26 15:15:22.529350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.529329 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:22.532572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.532548 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 26 15:15:22.532698 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.532612 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6jkj6\"" Jul 26 15:15:22.539602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.539577 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm"] Jul 26 15:15:22.667355 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.667226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mms4\" (UniqueName: \"kubernetes.io/projected/1c65057d-2fc3-464c-9afb-e6e51d9b83d8-kube-api-access-4mms4\") pod \"dns-operator-controller-manager-864dd78c4c-qxfzm\" (UID: \"1c65057d-2fc3-464c-9afb-e6e51d9b83d8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:22.767796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.767759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mms4\" (UniqueName: \"kubernetes.io/projected/1c65057d-2fc3-464c-9afb-e6e51d9b83d8-kube-api-access-4mms4\") pod \"dns-operator-controller-manager-864dd78c4c-qxfzm\" (UID: \"1c65057d-2fc3-464c-9afb-e6e51d9b83d8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:22.777724 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.777691 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mms4\" (UniqueName: \"kubernetes.io/projected/1c65057d-2fc3-464c-9afb-e6e51d9b83d8-kube-api-access-4mms4\") pod \"dns-operator-controller-manager-864dd78c4c-qxfzm\" (UID: \"1c65057d-2fc3-464c-9afb-e6e51d9b83d8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:22.840603 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.840558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:22.976122 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:22.976091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm"] Jul 26 15:15:22.978537 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:22.978513 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c65057d_2fc3_464c_9afb_e6e51d9b83d8.slice/crio-6a1a6e23ecb47389ddaec9766f396247ccb355dc896c92ca61505c30f0b8a416 WatchSource:0}: Error finding container 6a1a6e23ecb47389ddaec9766f396247ccb355dc896c92ca61505c30f0b8a416: Status 404 returned error can't find the container with id 6a1a6e23ecb47389ddaec9766f396247ccb355dc896c92ca61505c30f0b8a416 Jul 26 15:15:23.007773 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:23.007736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" event={"ID":"1c65057d-2fc3-464c-9afb-e6e51d9b83d8","Type":"ContainerStarted","Data":"6a1a6e23ecb47389ddaec9766f396247ccb355dc896c92ca61505c30f0b8a416"} Jul 26 15:15:23.009197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:23.009170 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" event={"ID":"3fb0e491-250d-4840-8da6-84203acbe577","Type":"ContainerStarted","Data":"72fbf5e6c2edc6ffe8bf0520a7c361610eba6f4017d1d31678355af1102f3895"} Jul 26 15:15:23.009333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:23.009325 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:23.048980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:23.048923 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" podStartSLOduration=1.886949982 podStartE2EDuration="4.048906896s" totalImagesPullingTime="2.161956914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:23.04821127 +0000 UTC m=+596.945886269" watchObservedRunningTime="2026-07-26 15:15:23.048906896 +0000 UTC m=+596.946581892" Jul 26 15:15:25.022200 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.022156 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" event={"ID":"1c65057d-2fc3-464c-9afb-e6e51d9b83d8","Type":"ContainerStarted","Data":"23f180a9852974bf358deaa3191d4fb5c94dbc3efd4c17847eb0221ad2589b5d"} Jul 26 15:15:25.022551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.022222 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:25.041977 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.041925 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" podStartSLOduration=1.066271932 podStartE2EDuration="3.041910542s" totalImagesPullingTime="1.97563861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:25.039864027 +0000 UTC m=+598.937539025" watchObservedRunningTime="2026-07-26 15:15:25.041910542 +0000 UTC m=+598.939585539" Jul 26 15:15:25.159425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.159386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh"] Jul 26 15:15:25.168847 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.168820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:25.171994 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.171970 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-46msh\"" Jul 26 15:15:25.173257 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.173231 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh"] Jul 26 15:15:25.292370 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.292269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hsc\" (UniqueName: \"kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc\") pod \"limitador-operator-controller-manager-859d7b78bd-749wh\" (UID: \"cdbad9bf-20fa-461e-b2f9-49034d1d9310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:25.393673 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.393632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hsc\" (UniqueName: \"kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc\") pod \"limitador-operator-controller-manager-859d7b78bd-749wh\" (UID: \"cdbad9bf-20fa-461e-b2f9-49034d1d9310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:25.403924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.403899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hsc\" (UniqueName: \"kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc\") pod \"limitador-operator-controller-manager-859d7b78bd-749wh\" (UID: \"cdbad9bf-20fa-461e-b2f9-49034d1d9310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:25.481341 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.481299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:25.630959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:25.630929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh"] Jul 26 15:15:25.632674 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:25.632644 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbad9bf_20fa_461e_b2f9_49034d1d9310.slice/crio-c36fc92bec6c704aa1d99c679e149cf9cb6756ae50177cda5d44e1a3bb1890cb WatchSource:0}: Error finding container c36fc92bec6c704aa1d99c679e149cf9cb6756ae50177cda5d44e1a3bb1890cb: Status 404 returned error can't find the container with id c36fc92bec6c704aa1d99c679e149cf9cb6756ae50177cda5d44e1a3bb1890cb Jul 26 15:15:26.028476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:26.028442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" event={"ID":"cdbad9bf-20fa-461e-b2f9-49034d1d9310","Type":"ContainerStarted","Data":"c36fc92bec6c704aa1d99c679e149cf9cb6756ae50177cda5d44e1a3bb1890cb"} Jul 26 15:15:28.044781 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:28.044745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" event={"ID":"cdbad9bf-20fa-461e-b2f9-49034d1d9310","Type":"ContainerStarted","Data":"ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f"} Jul 26 15:15:28.045182 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:28.044854 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:28.065802 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:28.065747 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" podStartSLOduration=1.267211908 podStartE2EDuration="3.065731119s" totalImagesPullingTime="1.798519211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:28.063432912 +0000 UTC m=+601.961107909" watchObservedRunningTime="2026-07-26 15:15:28.065731119 +0000 UTC m=+601.963406116" Jul 26 15:15:29.266189 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.266084 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk"] Jul 26 15:15:29.302790 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.269674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.302790 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.272534 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qkrp9\"" Jul 26 15:15:29.302790 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.285766 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk"] Jul 26 15:15:29.432774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.432738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44grm\" (UniqueName: \"kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.432951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.432806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.534007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.533916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.534007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.533985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44grm\" (UniqueName: \"kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.534396 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.534375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.542815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.542774 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44grm\" (UniqueName: \"kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.582465 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.582437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:29.720077 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:29.720053 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk"] Jul 26 15:15:29.722686 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:29.722654 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461b0813_f4cb_44e5_a88c_11bb5d1c859f.slice/crio-0e2f97f11bff835d773ef389eb25281a369dc45b529ff1793afa803151194f93 WatchSource:0}: Error finding container 0e2f97f11bff835d773ef389eb25281a369dc45b529ff1793afa803151194f93: Status 404 returned error can't find the container with id 0e2f97f11bff835d773ef389eb25281a369dc45b529ff1793afa803151194f93 Jul 26 15:15:30.058534 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:30.058492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" event={"ID":"461b0813-f4cb-44e5-a88c-11bb5d1c859f","Type":"ContainerStarted","Data":"0e2f97f11bff835d773ef389eb25281a369dc45b529ff1793afa803151194f93"} Jul 26 15:15:30.339068 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:30.338977 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:15:30.339495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:30.339405 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:15:34.018193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:34.018157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wh8hh" Jul 26 15:15:36.031137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:36.031106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qxfzm" Jul 26 15:15:36.096374 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:36.096327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" event={"ID":"461b0813-f4cb-44e5-a88c-11bb5d1c859f","Type":"ContainerStarted","Data":"380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd"} Jul 26 15:15:36.096537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:36.096431 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:36.114991 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:36.114943 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" podStartSLOduration=1.2472104 podStartE2EDuration="7.114928045s" totalImagesPullingTime="5.867717645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:36.112783882 +0000 UTC m=+610.010458890" watchObservedRunningTime="2026-07-26 15:15:36.114928045 +0000 UTC m=+610.012603105" Jul 26 15:15:39.053230 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:39.053196 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:47.104391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:47.104355 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:48.786704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.786664 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:15:48.789833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.789817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:48.794329 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.794304 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk"] Jul 26 15:15:48.794499 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.794478 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" containerName="manager" containerID="cri-o://380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd" gracePeriod=2 Jul 26 15:15:48.804220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.804192 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk"] Jul 26 15:15:48.807195 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.807167 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:15:48.821438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.821408 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh"] Jul 26 15:15:48.821627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.821606 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerName="manager" containerID="cri-o://ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f" gracePeriod=2 Jul 26 15:15:48.836049 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:48.836002 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:48.839206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.839182 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:15:48.839543 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.839530 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" containerName="manager" Jul 26 15:15:48.839591 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.839545 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" containerName="manager" Jul 26 15:15:48.839628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.839616 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" containerName="manager" Jul 26 15:15:48.842563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.842534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:48.845295 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.845255 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh"] Jul 26 15:15:48.847107 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:48.847081 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:48.857110 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.857080 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:15:48.875340 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:48.875295 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:48.901494 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.901460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:48.901661 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:48.901509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfl8\" (UniqueName: \"kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.002898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.002858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hn8g\" (UniqueName: \"kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.003060 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.003016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.003140 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.003067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfl8\" (UniqueName: \"kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.003140 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.003124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.003575 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.003550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.022127 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.022089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfl8\" (UniqueName: \"kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8\") pod \"kuadrant-operator-controller-manager-86878c769f-97hq6\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.065412 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.065382 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.068118 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.068090 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.068243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.068139 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.070672 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.070648 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.072965 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.072944 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.075738 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.075712 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.100979 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.100945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:49.103630 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.103606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.103710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.103656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hn8g\" (UniqueName: \"kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.104008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.103985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.115960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.115936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hn8g\" (UniqueName: \"kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hhmhk\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.162299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.162242 2575 generic.go:356] "Generic (PLEG): container finished" podID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerID="ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f" exitCode=0 Jul 26 15:15:49.162447 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.162314 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.162447 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.162432 2575 scope.go:117] "RemoveContainer" containerID="ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f" Jul 26 15:15:49.164950 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.164911 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.165429 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.165214 2575 generic.go:356] "Generic (PLEG): container finished" podID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" containerID="380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd" exitCode=0 Jul 26 15:15:49.165429 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.165270 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.167157 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.167007 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.169146 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.169113 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.171105 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.171053 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.172852 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.172751 2575 scope.go:117] "RemoveContainer" containerID="ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f" Jul 26 15:15:49.173060 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.173029 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f\": container with ID starting with ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f not found: ID does not exist" containerID="ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f" Jul 26 15:15:49.173107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.173057 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f"} err="failed to get container status \"ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f\": rpc error: code = NotFound desc = could not find container \"ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f\": container with ID starting with ebb6448ee2c6ea2e99c41d8c425190b9c385eadd24ce9f1e448d1291842ba05f not found: ID does not exist" Jul 26 15:15:49.173107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.173075 2575 scope.go:117] "RemoveContainer" containerID="380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd" Jul 26 15:15:49.181981 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.181952 2575 scope.go:117] "RemoveContainer" containerID="380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd" Jul 26 15:15:49.182340 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.182322 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd\": container with ID starting with 380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd not found: ID does not exist" containerID="380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd" Jul 26 15:15:49.182386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.182349 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd"} err="failed to get container status \"380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd\": rpc error: code = NotFound desc = could not find container \"380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd\": container with ID starting with 380ab0117efc6106f7ed9e37437b337bd69316a49f0a1d70455590b37003f6fd not found: ID does not exist" Jul 26 15:15:49.204691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.204655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume\") pod \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " Jul 26 15:15:49.204894 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.204730 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44grm\" (UniqueName: \"kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm\") pod \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\" (UID: \"461b0813-f4cb-44e5-a88c-11bb5d1c859f\") " Jul 26 15:15:49.204894 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.204769 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9hsc\" (UniqueName: \"kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc\") pod \"cdbad9bf-20fa-461e-b2f9-49034d1d9310\" (UID: \"cdbad9bf-20fa-461e-b2f9-49034d1d9310\") " Jul 26 15:15:49.205296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.205252 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "461b0813-f4cb-44e5-a88c-11bb5d1c859f" (UID: "461b0813-f4cb-44e5-a88c-11bb5d1c859f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:15:49.208346 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.208315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm" (OuterVolumeSpecName: "kube-api-access-44grm") pod "461b0813-f4cb-44e5-a88c-11bb5d1c859f" (UID: "461b0813-f4cb-44e5-a88c-11bb5d1c859f"). InnerVolumeSpecName "kube-api-access-44grm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:49.208470 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.208356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc" (OuterVolumeSpecName: "kube-api-access-n9hsc") pod "cdbad9bf-20fa-461e-b2f9-49034d1d9310" (UID: "cdbad9bf-20fa-461e-b2f9-49034d1d9310"). InnerVolumeSpecName "kube-api-access-n9hsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:15:49.231617 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.231582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:49.248784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.248755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:15:49.250858 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:49.250825 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7266d5_e64c_4a5a_b9ee_1bf5e3372457.slice/crio-476372bc68ea25d1a03fae98aaf6291edb48e0001b60546380e24df39b93d3c5 WatchSource:0}: Error finding container 476372bc68ea25d1a03fae98aaf6291edb48e0001b60546380e24df39b93d3c5: Status 404 returned error can't find the container with id 476372bc68ea25d1a03fae98aaf6291edb48e0001b60546380e24df39b93d3c5 Jul 26 15:15:49.305665 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.305588 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/461b0813-f4cb-44e5-a88c-11bb5d1c859f-extensions-socket-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:49.305665 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.305613 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44grm\" (UniqueName: \"kubernetes.io/projected/461b0813-f4cb-44e5-a88c-11bb5d1c859f-kube-api-access-44grm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:49.305665 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.305623 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9hsc\" (UniqueName: \"kubernetes.io/projected/cdbad9bf-20fa-461e-b2f9-49034d1d9310-kube-api-access-n9hsc\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:15:49.399815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:49.399785 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:15:49.402243 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:15:49.402218 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc9ab29_9a66_405c_9bc5_18a34dc85916.slice/crio-5a3625057edbd60d39bcd4f4236502b1053bbbed7228cb5116bbdc258f38aae6 WatchSource:0}: Error finding container 5a3625057edbd60d39bcd4f4236502b1053bbbed7228cb5116bbdc258f38aae6: Status 404 returned error can't find the container with id 5a3625057edbd60d39bcd4f4236502b1053bbbed7228cb5116bbdc258f38aae6 Jul 26 15:15:49.473444 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.473414 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.475506 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.475481 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:49.477512 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.477491 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:49.479438 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:49.479413 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:50.050945 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.050897 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:15:50.172523 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.172428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" event={"ID":"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457","Type":"ContainerStarted","Data":"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe"} Jul 26 15:15:50.172523 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.172479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" event={"ID":"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457","Type":"ContainerStarted","Data":"476372bc68ea25d1a03fae98aaf6291edb48e0001b60546380e24df39b93d3c5"} Jul 26 15:15:50.172733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.172523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:15:50.174116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.174079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" event={"ID":"8fc9ab29-9a66-405c-9bc5-18a34dc85916","Type":"ContainerStarted","Data":"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104"} Jul 26 15:15:50.174242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.174135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" event={"ID":"8fc9ab29-9a66-405c-9bc5-18a34dc85916","Type":"ContainerStarted","Data":"5a3625057edbd60d39bcd4f4236502b1053bbbed7228cb5116bbdc258f38aae6"} Jul 26 15:15:50.174242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.174165 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:15:50.196032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.195971 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" podStartSLOduration=2.19595484 podStartE2EDuration="2.19595484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:50.194369032 +0000 UTC m=+624.092044048" watchObservedRunningTime="2026-07-26 15:15:50.19595484 +0000 UTC m=+624.093629836" Jul 26 15:15:50.196521 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:50.196480 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:50.198447 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:50.198419 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:50.221632 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.221581 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" podStartSLOduration=2.221566628 podStartE2EDuration="2.221566628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:15:50.220325757 +0000 UTC m=+624.118000751" watchObservedRunningTime="2026-07-26 15:15:50.221566628 +0000 UTC m=+624.119241624" Jul 26 15:15:50.222499 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:50.222471 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-749wh\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-749wh" Jul 26 15:15:50.224407 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:15:50.224382 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6tgbk\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6tgbk" Jul 26 15:15:50.558336 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.558301 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="461b0813-f4cb-44e5-a88c-11bb5d1c859f" path="/var/lib/kubelet/pods/461b0813-f4cb-44e5-a88c-11bb5d1c859f/volumes" Jul 26 15:15:50.558623 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:15:50.558611 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" path="/var/lib/kubelet/pods/cdbad9bf-20fa-461e-b2f9-49034d1d9310/volumes" Jul 26 15:16:01.181790 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.181758 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:16:01.182318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.181936 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:16:01.266471 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.266430 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:16:01.266656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.266633 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" podUID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" containerName="manager" containerID="cri-o://5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104" gracePeriod=10 Jul 26 15:16:01.512096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.512070 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:16:01.603592 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.603566 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hn8g\" (UniqueName: \"kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g\") pod \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " Jul 26 15:16:01.603745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.603610 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume\") pod \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\" (UID: \"8fc9ab29-9a66-405c-9bc5-18a34dc85916\") " Jul 26 15:16:01.603997 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.603974 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8fc9ab29-9a66-405c-9bc5-18a34dc85916" (UID: "8fc9ab29-9a66-405c-9bc5-18a34dc85916"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:16:01.605718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.605696 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g" (OuterVolumeSpecName: "kube-api-access-7hn8g") pod "8fc9ab29-9a66-405c-9bc5-18a34dc85916" (UID: "8fc9ab29-9a66-405c-9bc5-18a34dc85916"). InnerVolumeSpecName "kube-api-access-7hn8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:01.704624 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.704535 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fc9ab29-9a66-405c-9bc5-18a34dc85916-extensions-socket-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:16:01.704624 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:01.704566 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hn8g\" (UniqueName: \"kubernetes.io/projected/8fc9ab29-9a66-405c-9bc5-18a34dc85916-kube-api-access-7hn8g\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:16:02.237944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.237911 2575 generic.go:356] "Generic (PLEG): container finished" podID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" containerID="5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104" exitCode=0 Jul 26 15:16:02.238392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.237962 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" Jul 26 15:16:02.238392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.237990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" event={"ID":"8fc9ab29-9a66-405c-9bc5-18a34dc85916","Type":"ContainerDied","Data":"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104"} Jul 26 15:16:02.238392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.238067 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk" event={"ID":"8fc9ab29-9a66-405c-9bc5-18a34dc85916","Type":"ContainerDied","Data":"5a3625057edbd60d39bcd4f4236502b1053bbbed7228cb5116bbdc258f38aae6"} Jul 26 15:16:02.238392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.238092 2575 scope.go:117] "RemoveContainer" containerID="5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104" Jul 26 15:16:02.250761 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.250159 2575 scope.go:117] "RemoveContainer" containerID="5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104" Jul 26 15:16:02.251396 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:16:02.251374 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104\": container with ID starting with 5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104 not found: ID does not exist" containerID="5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104" Jul 26 15:16:02.251455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.251407 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104"} err="failed to get container status \"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104\": rpc error: code = NotFound desc = could not find container \"5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104\": container with ID starting with 5d89a932f3fef23f0d492801cddaff9e2208d102a746342eb0647d4e660fb104 not found: ID does not exist" Jul 26 15:16:02.262878 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.262852 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:16:02.271045 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.271016 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hhmhk"] Jul 26 15:16:02.554225 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:02.554148 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" path="/var/lib/kubelet/pods/8fc9ab29-9a66-405c-9bc5-18a34dc85916/volumes" Jul 26 15:16:17.633343 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633302 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk"] Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633644 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerName="manager" Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633656 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerName="manager" Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633666 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" containerName="manager" Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633672 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" containerName="manager" Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633724 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc9ab29-9a66-405c-9bc5-18a34dc85916" containerName="manager" Jul 26 15:16:17.633745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.633733 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbad9bf-20fa-461e-b2f9-49034d1d9310" containerName="manager" Jul 26 15:16:17.636528 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.636505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.639605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.639582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d2rfq\"" Jul 26 15:16:17.640152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.640132 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 15:16:17.680173 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.680083 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk"] Jul 26 15:16:17.738826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.738826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.738996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jgh\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-kube-api-access-52jgh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.739069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.739092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.739160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.739127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840500 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840669 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840669 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jgh\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-kube-api-access-52jgh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.840895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.841110 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.840907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.841110 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.841028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.841209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.841129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.841404 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.841385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.843178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.843156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.843420 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.843402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.848838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.848812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.848955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.848943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jgh\" (UniqueName: \"kubernetes.io/projected/e8a474de-eaa9-4c6d-9a20-e8a442c86682-kube-api-access-52jgh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-pfbpk\" (UID: \"e8a474de-eaa9-4c6d-9a20-e8a442c86682\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:17.946449 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:17.946411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:18.101383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:18.101357 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk"] Jul 26 15:16:18.104291 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:16:18.104248 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a474de_eaa9_4c6d_9a20_e8a442c86682.slice/crio-942d1dcc0fdca72a6bd45b32f13e8b8f8de1016f54e6ec4f8f3b2b1656cf5d30 WatchSource:0}: Error finding container 942d1dcc0fdca72a6bd45b32f13e8b8f8de1016f54e6ec4f8f3b2b1656cf5d30: Status 404 returned error can't find the container with id 942d1dcc0fdca72a6bd45b32f13e8b8f8de1016f54e6ec4f8f3b2b1656cf5d30 Jul 26 15:16:18.106633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:18.106617 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:16:18.324227 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:18.324137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" event={"ID":"e8a474de-eaa9-4c6d-9a20-e8a442c86682","Type":"ContainerStarted","Data":"942d1dcc0fdca72a6bd45b32f13e8b8f8de1016f54e6ec4f8f3b2b1656cf5d30"} Jul 26 15:16:21.037224 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.037186 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:16:21.037515 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.037260 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:16:21.037515 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.037318 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:16:21.341562 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.341472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" event={"ID":"e8a474de-eaa9-4c6d-9a20-e8a442c86682","Type":"ContainerStarted","Data":"16f86e199f41b2ba1a3ce349dbdbe0834b908d65860c1213a3ab4b0e9c8c06ae"} Jul 26 15:16:21.363143 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.363091 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podStartSLOduration=1.432898949 podStartE2EDuration="4.36307524s" totalImagesPullingTime="2.930176291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:21.362033605 +0000 UTC m=+655.259708604" watchObservedRunningTime="2026-07-26 15:16:21.36307524 +0000 UTC m=+655.260750237" Jul 26 15:16:21.947134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:21.947098 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:22.314624 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.314542 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:22.318146 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.318127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.320816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.320791 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6r862\"" Jul 26 15:16:22.320931 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.320844 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 26 15:16:22.326046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.326025 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:22.387224 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.387191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.387424 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.387304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvk6\" (UniqueName: \"kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.418063 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.418027 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:22.487874 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.487837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvk6\" (UniqueName: \"kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.488085 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.488053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.488661 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.488641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.495990 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.495963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvk6\" (UniqueName: \"kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6\") pod \"limitador-limitador-64c94fd64f-9r5d4\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.624132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.624054 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:16:22.627952 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.627926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.629089 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.629066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:22.635373 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.635347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:16:22.665199 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.665165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:16:22.690156 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.690118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c17242eb-5d81-4075-8f5c-b15b59534c81-config-file\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.690319 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.690209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvdz\" (UniqueName: \"kubernetes.io/projected/c17242eb-5d81-4075-8f5c-b15b59534c81-kube-api-access-vbvdz\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.772649 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.772621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:22.774108 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:16:22.774081 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf094c3e_a330_43a1_82c0_b543d74ed384.slice/crio-782b49ca6d220b86ce8ea8104c8dd0417a87963c17a913089f0bb33362cb485d WatchSource:0}: Error finding container 782b49ca6d220b86ce8ea8104c8dd0417a87963c17a913089f0bb33362cb485d: Status 404 returned error can't find the container with id 782b49ca6d220b86ce8ea8104c8dd0417a87963c17a913089f0bb33362cb485d Jul 26 15:16:22.791633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.791603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c17242eb-5d81-4075-8f5c-b15b59534c81-config-file\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.791778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.791647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvdz\" (UniqueName: \"kubernetes.io/projected/c17242eb-5d81-4075-8f5c-b15b59534c81-kube-api-access-vbvdz\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.792444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.792424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c17242eb-5d81-4075-8f5c-b15b59534c81-config-file\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.799686 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.799655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvdz\" (UniqueName: \"kubernetes.io/projected/c17242eb-5d81-4075-8f5c-b15b59534c81-kube-api-access-vbvdz\") pod \"limitador-limitador-54d6d988d5-sv7ch\" (UID: \"c17242eb-5d81-4075-8f5c-b15b59534c81\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.941167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.941129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:22.952178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:22.952152 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:23.081493 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:23.081468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:16:23.083452 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:16:23.083421 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17242eb_5d81_4075_8f5c_b15b59534c81.slice/crio-cd1983f1c0a3354672310cfa04972cd7030e1d9fedc7eeefaabf420dca9d8439 WatchSource:0}: Error finding container cd1983f1c0a3354672310cfa04972cd7030e1d9fedc7eeefaabf420dca9d8439: Status 404 returned error can't find the container with id cd1983f1c0a3354672310cfa04972cd7030e1d9fedc7eeefaabf420dca9d8439 Jul 26 15:16:23.353738 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:23.353638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" event={"ID":"af094c3e-a330-43a1-82c0-b543d74ed384","Type":"ContainerStarted","Data":"782b49ca6d220b86ce8ea8104c8dd0417a87963c17a913089f0bb33362cb485d"} Jul 26 15:16:23.354831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:23.354804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" event={"ID":"c17242eb-5d81-4075-8f5c-b15b59534c81","Type":"ContainerStarted","Data":"cd1983f1c0a3354672310cfa04972cd7030e1d9fedc7eeefaabf420dca9d8439"} Jul 26 15:16:23.355116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:23.355103 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:23.356205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:23.356188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:16:26.375502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.375460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" event={"ID":"af094c3e-a330-43a1-82c0-b543d74ed384","Type":"ContainerStarted","Data":"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6"} Jul 26 15:16:26.375934 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.375529 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:26.377000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.376978 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" event={"ID":"c17242eb-5d81-4075-8f5c-b15b59534c81","Type":"ContainerStarted","Data":"d9f97e0184655e788aaf8d55077fe93dd61036872f5d86f18ea9702b0bf72cbf"} Jul 26 15:16:26.377080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.377037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:26.391650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.391598 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" podStartSLOduration=1.303450242 podStartE2EDuration="4.391584685s" totalImagesPullingTime="3.088134443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:26.390414263 +0000 UTC m=+660.288089290" watchObservedRunningTime="2026-07-26 15:16:26.391584685 +0000 UTC m=+660.289259682" Jul 26 15:16:26.405622 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:26.405576 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" podStartSLOduration=1.620292061 podStartE2EDuration="4.405562246s" totalImagesPullingTime="2.785270185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:26.405037868 +0000 UTC m=+660.302712864" watchObservedRunningTime="2026-07-26 15:16:26.405562246 +0000 UTC m=+660.303237242" Jul 26 15:16:37.382637 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:37.382606 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-sv7ch" Jul 26 15:16:37.383115 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:37.382672 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:37.446083 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:37.446053 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:37.446265 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:37.446232 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" podUID="af094c3e-a330-43a1-82c0-b543d74ed384" containerName="limitador" containerID="cri-o://66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6" gracePeriod=30 Jul 26 15:16:38.004632 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.004608 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:38.133208 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.133116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file\") pod \"af094c3e-a330-43a1-82c0-b543d74ed384\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " Jul 26 15:16:38.133376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.133215 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skvk6\" (UniqueName: \"kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6\") pod \"af094c3e-a330-43a1-82c0-b543d74ed384\" (UID: \"af094c3e-a330-43a1-82c0-b543d74ed384\") " Jul 26 15:16:38.133521 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.133498 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file" (OuterVolumeSpecName: "config-file") pod "af094c3e-a330-43a1-82c0-b543d74ed384" (UID: "af094c3e-a330-43a1-82c0-b543d74ed384"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:16:38.135427 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.135404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6" (OuterVolumeSpecName: "kube-api-access-skvk6") pod "af094c3e-a330-43a1-82c0-b543d74ed384" (UID: "af094c3e-a330-43a1-82c0-b543d74ed384"). InnerVolumeSpecName "kube-api-access-skvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:16:38.234631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.234594 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvk6\" (UniqueName: \"kubernetes.io/projected/af094c3e-a330-43a1-82c0-b543d74ed384-kube-api-access-skvk6\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:16:38.234631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.234626 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af094c3e-a330-43a1-82c0-b543d74ed384-config-file\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:16:38.441769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.441729 2575 generic.go:356] "Generic (PLEG): container finished" podID="af094c3e-a330-43a1-82c0-b543d74ed384" containerID="66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6" exitCode=0 Jul 26 15:16:38.441769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.441773 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" Jul 26 15:16:38.442358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.441811 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" event={"ID":"af094c3e-a330-43a1-82c0-b543d74ed384","Type":"ContainerDied","Data":"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6"} Jul 26 15:16:38.442358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.441871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-9r5d4" event={"ID":"af094c3e-a330-43a1-82c0-b543d74ed384","Type":"ContainerDied","Data":"782b49ca6d220b86ce8ea8104c8dd0417a87963c17a913089f0bb33362cb485d"} Jul 26 15:16:38.442358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.441894 2575 scope.go:117] "RemoveContainer" containerID="66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6" Jul 26 15:16:38.453463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.453224 2575 scope.go:117] "RemoveContainer" containerID="66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6" Jul 26 15:16:38.453557 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:16:38.453525 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6\": container with ID starting with 66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6 not found: ID does not exist" containerID="66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6" Jul 26 15:16:38.453615 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.453560 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6"} err="failed to get container status \"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6\": rpc error: code = NotFound desc = could not find container \"66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6\": container with ID starting with 66c27d94537f9e488f2ee38e5c3d84786a1df13260336b3baf555ffd7fc8f9e6 not found: ID does not exist" Jul 26 15:16:38.465335 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.465303 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:38.467545 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.467525 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-9r5d4"] Jul 26 15:16:38.554125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.554093 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af094c3e-a330-43a1-82c0-b543d74ed384" path="/var/lib/kubelet/pods/af094c3e-a330-43a1-82c0-b543d74ed384/volumes" Jul 26 15:16:38.889402 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.889328 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jzg62"] Jul 26 15:16:38.889679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.889667 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af094c3e-a330-43a1-82c0-b543d74ed384" containerName="limitador" Jul 26 15:16:38.889721 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.889681 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af094c3e-a330-43a1-82c0-b543d74ed384" containerName="limitador" Jul 26 15:16:38.889764 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.889755 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af094c3e-a330-43a1-82c0-b543d74ed384" containerName="limitador" Jul 26 15:16:38.894339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.894317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:38.897100 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.897041 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 26 15:16:38.898065 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.898041 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-srwz5\"" Jul 26 15:16:38.898197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.898068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 26 15:16:38.898197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.898096 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:16:38.899816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:38.899793 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jzg62"] Jul 26 15:16:39.042069 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.042032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894qt\" (UniqueName: \"kubernetes.io/projected/dbc3428a-481c-4dfc-883a-e2175455b907-kube-api-access-894qt\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.042246 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.042082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3428a-481c-4dfc-883a-e2175455b907-data\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.143448 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.143358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3428a-481c-4dfc-883a-e2175455b907-data\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.143592 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.143453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894qt\" (UniqueName: \"kubernetes.io/projected/dbc3428a-481c-4dfc-883a-e2175455b907-kube-api-access-894qt\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.143770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.143752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dbc3428a-481c-4dfc-883a-e2175455b907-data\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.152196 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.152168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894qt\" (UniqueName: \"kubernetes.io/projected/dbc3428a-481c-4dfc-883a-e2175455b907-kube-api-access-894qt\") pod \"postgres-69fb4594cf-jzg62\" (UID: \"dbc3428a-481c-4dfc-883a-e2175455b907\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.206123 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.206086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:39.341054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.341028 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jzg62"] Jul 26 15:16:39.343055 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:16:39.343027 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc3428a_481c_4dfc_883a_e2175455b907.slice/crio-309673a57db5de1f7aeecc6b544f203cd4a0a029b7b0591c8dd48e71ab81c97e WatchSource:0}: Error finding container 309673a57db5de1f7aeecc6b544f203cd4a0a029b7b0591c8dd48e71ab81c97e: Status 404 returned error can't find the container with id 309673a57db5de1f7aeecc6b544f203cd4a0a029b7b0591c8dd48e71ab81c97e Jul 26 15:16:39.448550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:39.448519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" event={"ID":"dbc3428a-481c-4dfc-883a-e2175455b907","Type":"ContainerStarted","Data":"309673a57db5de1f7aeecc6b544f203cd4a0a029b7b0591c8dd48e71ab81c97e"} Jul 26 15:16:44.482081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:44.482040 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" event={"ID":"dbc3428a-481c-4dfc-883a-e2175455b907","Type":"ContainerStarted","Data":"72d106fdeb649a58027480d39d50e10eae8854cabf5a1807a0f4ca580b97602b"} Jul 26 15:16:44.482480 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:44.482142 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:16:44.498395 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:44.498345 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" podStartSLOduration=1.5293223839999999 podStartE2EDuration="6.498326231s" totalImagesPullingTime="4.969003847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:16:44.49656725 +0000 UTC m=+678.394242249" watchObservedRunningTime="2026-07-26 15:16:44.498326231 +0000 UTC m=+678.396001229" Jul 26 15:16:50.516841 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:16:50.516807 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jzg62" Jul 26 15:17:30.985990 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:30.985953 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv"] Jul 26 15:17:30.990746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:30.990730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:30.993478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:30.993449 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xgdzp\"" Jul 26 15:17:30.993602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:30.993578 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 26 15:17:30.998158 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:30.998135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv"] Jul 26 15:17:31.006305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.006251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc48ed-b26c-41be-898a-478c34b4651f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.006423 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.006360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nfg\" (UniqueName: \"kubernetes.io/projected/2bfc48ed-b26c-41be-898a-478c34b4651f-kube-api-access-z8nfg\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.107175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.107140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nfg\" (UniqueName: \"kubernetes.io/projected/2bfc48ed-b26c-41be-898a-478c34b4651f-kube-api-access-z8nfg\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.107386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.107233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc48ed-b26c-41be-898a-478c34b4651f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.107989 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.107971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc48ed-b26c-41be-898a-478c34b4651f-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.116115 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.116089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nfg\" (UniqueName: \"kubernetes.io/projected/2bfc48ed-b26c-41be-898a-478c34b4651f-kube-api-access-z8nfg\") pod \"payload-pre-processing-687dbb6cf6-68zjv\" (UID: \"2bfc48ed-b26c-41be-898a-478c34b4651f\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.131832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.131798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:17:31.303376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.303265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:31.447515 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.447489 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv"] Jul 26 15:17:31.449414 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:17:31.449391 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfc48ed_b26c_41be_898a_478c34b4651f.slice/crio-5c382171750e1bbbe48c3d19adf98dbaf1296219d5fe80a812d15241414fde32 WatchSource:0}: Error finding container 5c382171750e1bbbe48c3d19adf98dbaf1296219d5fe80a812d15241414fde32: Status 404 returned error can't find the container with id 5c382171750e1bbbe48c3d19adf98dbaf1296219d5fe80a812d15241414fde32 Jul 26 15:17:31.723886 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:31.723847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" event={"ID":"2bfc48ed-b26c-41be-898a-478c34b4651f","Type":"ContainerStarted","Data":"5c382171750e1bbbe48c3d19adf98dbaf1296219d5fe80a812d15241414fde32"} Jul 26 15:17:33.737166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:33.737124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" event={"ID":"2bfc48ed-b26c-41be-898a-478c34b4651f","Type":"ContainerStarted","Data":"68de4295f88f9e0bb44e59fd10a661e8f31a9bd1b1f70396d8c12874bc5ae2ff"} Jul 26 15:17:33.737645 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:33.737304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:33.753829 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:33.753775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" podStartSLOduration=1.952102759 podStartE2EDuration="3.753759938s" totalImagesPullingTime="1.801657179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:17:33.751382349 +0000 UTC m=+727.649057370" watchObservedRunningTime="2026-07-26 15:17:33.753759938 +0000 UTC m=+727.651434935" Jul 26 15:17:38.458185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.458141 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pfbpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:40038->10.134.0.39:15021: read: connection reset by peer" start-of-body= Jul 26 15:17:38.458651 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.458200 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podUID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:40038->10.134.0.39:15021: read: connection reset by peer" Jul 26 15:17:38.766637 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.766541 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerID="16f86e199f41b2ba1a3ce349dbdbe0834b908d65860c1213a3ab4b0e9c8c06ae" exitCode=0 Jul 26 15:17:38.766637 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.766622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" event={"ID":"e8a474de-eaa9-4c6d-9a20-e8a442c86682","Type":"ContainerDied","Data":"16f86e199f41b2ba1a3ce349dbdbe0834b908d65860c1213a3ab4b0e9c8c06ae"} Jul 26 15:17:38.767052 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.767037 2575 scope.go:117] "RemoveContainer" containerID="16f86e199f41b2ba1a3ce349dbdbe0834b908d65860c1213a3ab4b0e9c8c06ae" Jul 26 15:17:38.767998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.767971 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:17:38.768112 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.768043 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:17:38.768112 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:38.768073 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:17:39.774128 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:39.774096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" event={"ID":"e8a474de-eaa9-4c6d-9a20-e8a442c86682","Type":"ContainerStarted","Data":"66651f293c1077e4faf8ac3975b142349861cb3fba75ea59b2ecdb28bf486eff"} Jul 26 15:17:39.947607 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:39.947571 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:17:39.947803 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:39.947781 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pfbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:17:39.947855 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:39.947830 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podUID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 15:17:40.946809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:40.946767 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pfbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:17:40.947193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:40.946828 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podUID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 15:17:41.947466 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:41.947431 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pfbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:17:41.947861 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:41.947500 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podUID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 15:17:42.947172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:42.947135 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-pfbpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:17:42.947379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:42.947187 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" podUID="e8a474de-eaa9-4c6d-9a20-e8a442c86682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 15:17:43.950174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:43.950143 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:17:43.950595 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:43.950507 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:17:43.951239 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:43.951221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-pfbpk" Jul 26 15:17:44.745214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:44.745167 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" podUID="2bfc48ed-b26c-41be-898a-478c34b4651f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.43:9004: connect: connection refused" Jul 26 15:17:54.744084 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:54.744053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-68zjv" Jul 26 15:17:59.442093 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:17:59.442052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:18:04.225437 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.225347 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr"] Jul 26 15:18:04.229104 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.229084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.232905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.232881 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ks67c\"" Jul 26 15:18:04.233043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.232980 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 15:18:04.233043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.233025 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 26 15:18:04.233146 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.233051 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 15:18:04.238771 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.238749 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr"] Jul 26 15:18:04.419749 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.419937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.419937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.419937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.419937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqxb\" (UniqueName: \"kubernetes.io/projected/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-kube-api-access-kzqxb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.420167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.419959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521102 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521102 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqxb\" (UniqueName: \"kubernetes.io/projected/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-kube-api-access-kzqxb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521674 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521794 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.521881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.521860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.523606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.523576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.523876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.523858 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.529494 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.529466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqxb\" (UniqueName: \"kubernetes.io/projected/74c621f9-1ad4-4260-aff0-e57ecc72b0ea-kube-api-access-kzqxb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4nrr\" (UID: \"74c621f9-1ad4-4260-aff0-e57ecc72b0ea\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.541167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.541142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:04.679267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.679240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr"] Jul 26 15:18:04.682251 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:18:04.682223 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c621f9_1ad4_4260_aff0_e57ecc72b0ea.slice/crio-baf1fd2dd671035aa9297a04db556d8ad60b0af7991a223f0340c3ae7f3a818f WatchSource:0}: Error finding container baf1fd2dd671035aa9297a04db556d8ad60b0af7991a223f0340c3ae7f3a818f: Status 404 returned error can't find the container with id baf1fd2dd671035aa9297a04db556d8ad60b0af7991a223f0340c3ae7f3a818f Jul 26 15:18:04.913404 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:04.913320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" event={"ID":"74c621f9-1ad4-4260-aff0-e57ecc72b0ea","Type":"ContainerStarted","Data":"baf1fd2dd671035aa9297a04db556d8ad60b0af7991a223f0340c3ae7f3a818f"} Jul 26 15:18:05.238010 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:05.237371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:18:05.920328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:05.920210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" event={"ID":"74c621f9-1ad4-4260-aff0-e57ecc72b0ea","Type":"ContainerStarted","Data":"b0c4327b2f54b921fc9a39954e4d4fbbca875a371691e4048c0d0cb73fb9359d"} Jul 26 15:18:05.938797 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:05.938748 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" podStartSLOduration=0.958287418 podStartE2EDuration="1.938723896s" totalImagesPullingTime="980.436478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:05.937142636 +0000 UTC m=+759.834817633" watchObservedRunningTime="2026-07-26 15:18:05.938723896 +0000 UTC m=+759.836398893" Jul 26 15:18:14.542155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:14.542117 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:14.542578 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:14.542184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:14.555335 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:14.555307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:14.980643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:14.980612 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4nrr" Jul 26 15:18:24.042928 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:24.042893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:18:37.044704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:37.044666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:18:59.944492 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:18:59.944454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:19:12.230881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:19:12.230844 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:20:30.380024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:20:30.379988 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:20:30.381067 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:20:30.381041 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:20:56.035338 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:20:56.035304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:21:05.234557 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:21:05.234516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:21:16.028746 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:21:16.028705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:22:58.930988 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:22:58.930948 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:23:08.345807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:23:08.345757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:23:38.140348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:23:38.140304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:23:48.032251 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:23:48.032211 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:23:58.643835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:23:58.643791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:24:06.260265 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:24:06.260230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:24:19.134576 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:24:19.134543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:24:29.689370 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:24:29.689331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:24:40.133854 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:24:40.133821 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:24:49.934408 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:24:49.934323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:25:12.529544 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:25:12.529503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:25:24.730693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:25:24.730658 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:25:30.416043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:25:30.416014 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:25:30.417971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:25:30.417943 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:25:47.344900 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:25:47.344856 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:01.046295 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:01.046232 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:13.243439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:13.243399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:22.834413 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:22.834325 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:33.633473 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:33.633434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:43.527174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:43.527135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:26:52.630038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:26:52.629993 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:27:17.619635 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:17.619599 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:27:17.620206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:17.619843 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" podUID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" containerName="manager" containerID="cri-o://651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe" gracePeriod=10 Jul 26 15:27:20.884369 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.884339 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:27:20.921380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.921349 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" containerID="651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe" exitCode=0 Jul 26 15:27:20.921586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.921394 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" Jul 26 15:27:20.921586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.921438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" event={"ID":"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457","Type":"ContainerDied","Data":"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe"} Jul 26 15:27:20.921586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.921509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6" event={"ID":"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457","Type":"ContainerDied","Data":"476372bc68ea25d1a03fae98aaf6291edb48e0001b60546380e24df39b93d3c5"} Jul 26 15:27:20.921586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.921529 2575 scope.go:117] "RemoveContainer" containerID="651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe" Jul 26 15:27:20.933542 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.933516 2575 scope.go:117] "RemoveContainer" containerID="651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe" Jul 26 15:27:20.933877 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:27:20.933852 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe\": container with ID starting with 651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe not found: ID does not exist" containerID="651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe" Jul 26 15:27:20.933992 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:20.933885 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe"} err="failed to get container status \"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe\": rpc error: code = NotFound desc = could not find container \"651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe\": container with ID starting with 651e5335de45251b59b31af16ed1952d23fb82012c2d4ca4be5ee42c8e9f62fe not found: ID does not exist" Jul 26 15:27:21.011877 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.011844 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjfl8\" (UniqueName: \"kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8\") pod \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " Jul 26 15:27:21.012076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.011949 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume\") pod \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\" (UID: \"9d7266d5-e64c-4a5a-b9ee-1bf5e3372457\") " Jul 26 15:27:21.012349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.012325 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" (UID: "9d7266d5-e64c-4a5a-b9ee-1bf5e3372457"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:27:21.014140 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.014117 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8" (OuterVolumeSpecName: "kube-api-access-zjfl8") pod "9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" (UID: "9d7266d5-e64c-4a5a-b9ee-1bf5e3372457"). InnerVolumeSpecName "kube-api-access-zjfl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:27:21.112856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.112801 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-extensions-socket-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:27:21.112856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.112848 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfl8\" (UniqueName: \"kubernetes.io/projected/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457-kube-api-access-zjfl8\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:27:21.245333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.245301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:27:21.250209 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:21.250178 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-97hq6"] Jul 26 15:27:22.554076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:27:22.554039 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" path="/var/lib/kubelet/pods/9d7266d5-e64c-4a5a-b9ee-1bf5e3372457/volumes" Jul 26 15:28:18.782464 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.782379 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb"] Jul 26 15:28:18.784905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.782778 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" containerName="manager" Jul 26 15:28:18.784905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.782791 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" containerName="manager" Jul 26 15:28:18.784905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.782871 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d7266d5-e64c-4a5a-b9ee-1bf5e3372457" containerName="manager" Jul 26 15:28:18.785737 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.785721 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:18.788490 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.788467 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qkrp9\"" Jul 26 15:28:18.797956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.797929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb"] Jul 26 15:28:18.908540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.908494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1a348c3-73b2-48af-83e6-af8db0a6ee62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:18.908730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:18.908553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psljf\" (UniqueName: \"kubernetes.io/projected/f1a348c3-73b2-48af-83e6-af8db0a6ee62-kube-api-access-psljf\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.009958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.009913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1a348c3-73b2-48af-83e6-af8db0a6ee62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.010155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.009969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psljf\" (UniqueName: \"kubernetes.io/projected/f1a348c3-73b2-48af-83e6-af8db0a6ee62-kube-api-access-psljf\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.010450 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.010412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1a348c3-73b2-48af-83e6-af8db0a6ee62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.020236 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.020214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psljf\" (UniqueName: \"kubernetes.io/projected/f1a348c3-73b2-48af-83e6-af8db0a6ee62-kube-api-access-psljf\") pod \"kuadrant-operator-controller-manager-86878c769f-k2wjb\" (UID: \"f1a348c3-73b2-48af-83e6-af8db0a6ee62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.096715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.096615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:19.240697 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:19.240671 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb"] Jul 26 15:28:19.242783 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:28:19.242751 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a348c3_73b2_48af_83e6_af8db0a6ee62.slice/crio-9ce3d8afa1a1dd229f77d64d4c69982d56f289d5cd14d950894ff544fb467335 WatchSource:0}: Error finding container 9ce3d8afa1a1dd229f77d64d4c69982d56f289d5cd14d950894ff544fb467335: Status 404 returned error can't find the container with id 9ce3d8afa1a1dd229f77d64d4c69982d56f289d5cd14d950894ff544fb467335 Jul 26 15:28:20.239343 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:20.239301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" event={"ID":"f1a348c3-73b2-48af-83e6-af8db0a6ee62","Type":"ContainerStarted","Data":"a305548348a2cbd48bd9ba4370efb6f876fce08226a6f520aeb0b2659e6f40b6"} Jul 26 15:28:20.239810 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:20.239372 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:20.239810 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:20.239397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" event={"ID":"f1a348c3-73b2-48af-83e6-af8db0a6ee62","Type":"ContainerStarted","Data":"9ce3d8afa1a1dd229f77d64d4c69982d56f289d5cd14d950894ff544fb467335"} Jul 26 15:28:20.264111 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:20.264062 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" podStartSLOduration=2.264047008 podStartE2EDuration="2.264047008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:28:20.261399875 +0000 UTC m=+1374.159074890" watchObservedRunningTime="2026-07-26 15:28:20.264047008 +0000 UTC m=+1374.161722006" Jul 26 15:28:31.246831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:31.246795 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k2wjb" Jul 26 15:28:38.441801 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:38.441762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:28:48.126652 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:28:48.126615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:01.441729 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:01.441692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:10.633744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:10.633709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:19.026699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:19.026598 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:28.729583 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:28.729543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:39.632614 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:39.632563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:50.532532 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:50.532488 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:29:59.938068 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:29:59.938026 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:30:00.150202 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.150158 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts"] Jul 26 15:30:00.156089 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.156050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" Jul 26 15:30:00.159169 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.159133 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:30:00.160511 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.160481 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts"] Jul 26 15:30:00.171955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.171929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" Jul 26 15:30:00.353693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.353648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts"] Jul 26 15:30:00.373835 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:30:00.373795 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e499fa_3191_4755_be03_4a125134604c.slice/crio-262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9 WatchSource:0}: Error finding container 262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9: Status 404 returned error can't find the container with id 262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9 Jul 26 15:30:00.376142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.376120 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:30:00.774967 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:00.774931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" event={"ID":"c3e499fa-3191-4755-be03-4a125134604c","Type":"ContainerStarted","Data":"262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9"} Jul 26 15:30:11.523982 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:11.523940 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:30:22.237383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:22.237348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:30:25.921188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:25.921144 2575 generic.go:356] "Generic (PLEG): container finished" podID="c3e499fa-3191-4755-be03-4a125134604c" containerID="30faef4ca3fb2bd8b9de95424d8198874937daa9cc56967b1fdd1d83a665d3c9" exitCode=0 Jul 26 15:30:25.921823 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:25.921231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" event={"ID":"c3e499fa-3191-4755-be03-4a125134604c","Type":"ContainerDied","Data":"30faef4ca3fb2bd8b9de95424d8198874937daa9cc56967b1fdd1d83a665d3c9"} Jul 26 15:30:27.066517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:27.066489 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" Jul 26 15:30:27.933895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:27.933854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" event={"ID":"c3e499fa-3191-4755-be03-4a125134604c","Type":"ContainerDied","Data":"262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9"} Jul 26 15:30:27.934078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:27.933914 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262864965e025e6e08c6d6547c564a20e378c94d33f25eea0ca6c0c77e7b6af9" Jul 26 15:30:27.934078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:27.933880 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-ltcts" Jul 26 15:30:30.450429 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:30.450398 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:30:30.453854 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:30.453831 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:30:32.037138 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:32.037104 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:30:52.137450 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:30:52.137359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:31:00.333194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:31:00.333154 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:31:16.228505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:31:16.228466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:31:44.028055 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:31:44.028017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:31:57.740459 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:31:57.740421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:32:04.433965 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:32:04.433923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:32:21.527601 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:32:21.527517 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:32:30.833175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:32:30.833139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:32:40.146208 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:32:40.146164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:32:49.337908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:32:49.337873 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:00.342509 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:00.342468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:09.234668 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:09.234630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:22.833211 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:22.833170 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:31.841959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:31.841921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:41.533365 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:41.533328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:33:51.830804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:33:51.830711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:03.035150 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:03.035113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:09.337248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:09.337208 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:25.435032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:25.434996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:36.137706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:36.137665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:45.233535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:45.233495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:34:53.728311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:34:53.728258 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:35:02.540295 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:02.540246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:35:25.043439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:25.043402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:35:29.751013 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.750976 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:35:29.751458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.751411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e499fa-3191-4755-be03-4a125134604c" containerName="cleanup" Jul 26 15:35:29.751458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.751424 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e499fa-3191-4755-be03-4a125134604c" containerName="cleanup" Jul 26 15:35:29.751536 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.751488 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e499fa-3191-4755-be03-4a125134604c" containerName="cleanup" Jul 26 15:35:29.754403 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.754388 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.756944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.756922 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-50ce075b-openshift-default-dockercfg-nlh7t\"" Jul 26 15:35:29.769352 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgj8\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769501 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769501 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769543 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.769766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.769753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.774732 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.774707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:35:29.870568 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgj8\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870568 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.870985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871141 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.870985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871204 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.871166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871264 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.871196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.871330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871423 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.871401 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.871656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.871638 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.873257 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.873236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.873466 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.873445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.880380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.880347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgj8\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:29.880380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:29.880377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token\") pod \"e2e-ait-50ce075b-openshift-default-5986f687d6-986fl\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:30.064657 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.064568 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:30.209769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.209740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:35:30.211051 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:35:30.211018 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa39021_cd12_4fce_bed5_59b852bc4d0e.slice/crio-7b721fc5de852c76fa37241dc6cbd3598fe78842fd4da05365739e9c75088ba3 WatchSource:0}: Error finding container 7b721fc5de852c76fa37241dc6cbd3598fe78842fd4da05365739e9c75088ba3: Status 404 returned error can't find the container with id 7b721fc5de852c76fa37241dc6cbd3598fe78842fd4da05365739e9c75088ba3 Jul 26 15:35:30.213301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.213240 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:30.213392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.213344 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:30.213436 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.213392 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:30.492780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.492749 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:35:30.497735 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.497708 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:35:30.546553 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.546517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerStarted","Data":"d8c37d17ece91e2f63ee1255ebcbfa9508b497ac27e1f638cb9a6a7fa409d592"} Jul 26 15:35:30.546742 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.546578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerStarted","Data":"7b721fc5de852c76fa37241dc6cbd3598fe78842fd4da05365739e9c75088ba3"} Jul 26 15:35:30.567391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:30.567337 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podStartSLOduration=1.567317127 podStartE2EDuration="1.567317127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:35:30.565214926 +0000 UTC m=+1804.462889924" watchObservedRunningTime="2026-07-26 15:35:30.567317127 +0000 UTC m=+1804.464992125" Jul 26 15:35:31.065098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:31.065060 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:31.070133 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:31.070104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:31.552170 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:31.552134 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:31.553298 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:31.553251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:32.677262 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.677217 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:35:32.681062 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.681038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.689174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.689151 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50ce075b\"" Jul 26 15:35:32.689327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.689184 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50ce075b-dockercfg-jlltw\"" Jul 26 15:35:32.693444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.693422 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:35:32.698641 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.698614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lkwx\" (UniqueName: \"kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.698761 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.698675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.799661 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.799625 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lkwx\" (UniqueName: \"kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.799834 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.799675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.800370 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.800350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.807465 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.807440 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lkwx\" (UniqueName: \"kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx\") pod \"payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:32.993298 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:32.993185 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:33.137754 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:33.137726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:35:33.140128 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:35:33.140101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ee0cd9_3e1e_43b8_8607_f3d827149245.slice/crio-358bf02a0a3d5df817267caac3119d515f16330963641b728f49733aa549dd69 WatchSource:0}: Error finding container 358bf02a0a3d5df817267caac3119d515f16330963641b728f49733aa549dd69: Status 404 returned error can't find the container with id 358bf02a0a3d5df817267caac3119d515f16330963641b728f49733aa549dd69 Jul 26 15:35:33.564554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:33.564516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" event={"ID":"71ee0cd9-3e1e-43b8-8607-f3d827149245","Type":"ContainerStarted","Data":"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b"} Jul 26 15:35:33.564832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:33.564573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" event={"ID":"71ee0cd9-3e1e-43b8-8607-f3d827149245","Type":"ContainerStarted","Data":"358bf02a0a3d5df817267caac3119d515f16330963641b728f49733aa549dd69"} Jul 26 15:35:33.564832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:33.564770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:33.581643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:33.581590 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" podStartSLOduration=1.581574249 podStartE2EDuration="1.581574249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:35:33.579927962 +0000 UTC m=+1807.477602964" watchObservedRunningTime="2026-07-26 15:35:33.581574249 +0000 UTC m=+1807.479249247" Jul 26 15:35:36.328769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:36.328731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:35:44.572355 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.572248 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:35:44.640527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.640495 2575 generic.go:356] "Generic (PLEG): container finished" podID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerID="d8c37d17ece91e2f63ee1255ebcbfa9508b497ac27e1f638cb9a6a7fa409d592" exitCode=0 Jul 26 15:35:44.640684 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.640571 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerDied","Data":"d8c37d17ece91e2f63ee1255ebcbfa9508b497ac27e1f638cb9a6a7fa409d592"} Jul 26 15:35:44.640964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.640951 2575 scope.go:117] "RemoveContainer" containerID="d8c37d17ece91e2f63ee1255ebcbfa9508b497ac27e1f638cb9a6a7fa409d592" Jul 26 15:35:44.641998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.641968 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:44.642103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.642055 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:44.642205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:44.642105 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:35:45.649175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:45.649139 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerStarted","Data":"5eb02844bcf89c83acac4801a38f27553d906bedec3245dd727fbb134c0ce7b1"} Jul 26 15:35:46.065319 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:46.065260 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:46.066224 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:46.066197 2575 patch_prober.go:28] interesting pod/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:35:46.066347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:46.066246 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 15:35:46.561485 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:46.561455 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:47.065006 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:47.064970 2575 patch_prober.go:28] interesting pod/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:35:47.065454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:47.065029 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 15:35:48.065243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:48.065202 2575 patch_prober.go:28] interesting pod/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:35:48.065664 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:48.065313 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 15:35:49.065405 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:49.065365 2575 patch_prober.go:28] interesting pod/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:35:49.065793 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:49.065433 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 15:35:50.069386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:50.069349 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:50.070426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:50.070405 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:35:57.463119 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.463056 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2"] Jul 26 15:35:57.466626 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.466608 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:57.469069 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.469052 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:35:57.474569 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.474540 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2"] Jul 26 15:35:57.633750 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.633709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ce075b-k96g2\" (UID: \"b60e8084-ad4a-4a85-95be-ed47f6340746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:57.734715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.734631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ce075b-k96g2\" (UID: \"b60e8084-ad4a-4a85-95be-ed47f6340746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:57.735292 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.735257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50ce075b-k96g2\" (UID: \"b60e8084-ad4a-4a85-95be-ed47f6340746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:57.779663 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.779613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:57.914706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:57.914679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2"] Jul 26 15:35:57.916332 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:35:57.916305 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e8084_ad4a_4a85_95be_ed47f6340746.slice/crio-216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8 WatchSource:0}: Error finding container 216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8: Status 404 returned error can't find the container with id 216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8 Jul 26 15:35:58.719435 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:58.719398 2575 generic.go:356] "Generic (PLEG): container finished" podID="b60e8084-ad4a-4a85-95be-ed47f6340746" containerID="53103554c4fd3861baab93ebc25966bf07cc1069233339d40d88b27597233021" exitCode=0 Jul 26 15:35:58.719815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:58.719449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" event={"ID":"b60e8084-ad4a-4a85-95be-ed47f6340746","Type":"ContainerDied","Data":"53103554c4fd3861baab93ebc25966bf07cc1069233339d40d88b27597233021"} Jul 26 15:35:58.719815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:58.719504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" event={"ID":"b60e8084-ad4a-4a85-95be-ed47f6340746","Type":"ContainerStarted","Data":"216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8"} Jul 26 15:35:59.861691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:59.861663 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:35:59.952689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:59.952656 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca\") pod \"b60e8084-ad4a-4a85-95be-ed47f6340746\" (UID: \"b60e8084-ad4a-4a85-95be-ed47f6340746\") " Jul 26 15:35:59.953028 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:35:59.953002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b60e8084-ad4a-4a85-95be-ed47f6340746" (UID: "b60e8084-ad4a-4a85-95be-ed47f6340746"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:00.053650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:00.053544 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60e8084-ad4a-4a85-95be-ed47f6340746-maas-api-service-ca\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:00.733768 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:00.733731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" event={"ID":"b60e8084-ad4a-4a85-95be-ed47f6340746","Type":"ContainerDied","Data":"216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8"} Jul 26 15:36:00.733951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:00.733785 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="216fdf92e65b5c2a087f262a3b3864a936db527bb95f1183990c98fefe9a1cd8" Jul 26 15:36:00.733951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:00.733747 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2" Jul 26 15:36:07.935056 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:07.935016 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:36:07.935671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:07.935306 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" podUID="71ee0cd9-3e1e-43b8-8607-f3d827149245" containerName="payload-pre-processing" containerID="cri-o://4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b" gracePeriod=30 Jul 26 15:36:08.200873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.200794 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:36:08.227963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.227926 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume\") pod \"71ee0cd9-3e1e-43b8-8607-f3d827149245\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " Jul 26 15:36:08.228136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.227990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lkwx\" (UniqueName: \"kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx\") pod \"71ee0cd9-3e1e-43b8-8607-f3d827149245\" (UID: \"71ee0cd9-3e1e-43b8-8607-f3d827149245\") " Jul 26 15:36:08.228401 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.228376 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71ee0cd9-3e1e-43b8-8607-f3d827149245" (UID: "71ee0cd9-3e1e-43b8-8607-f3d827149245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:08.230296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.230256 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx" (OuterVolumeSpecName: "kube-api-access-6lkwx") pod "71ee0cd9-3e1e-43b8-8607-f3d827149245" (UID: "71ee0cd9-3e1e-43b8-8607-f3d827149245"). InnerVolumeSpecName "kube-api-access-6lkwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:36:08.329655 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.329615 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ee0cd9-3e1e-43b8-8607-f3d827149245-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:08.329655 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.329646 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lkwx\" (UniqueName: \"kubernetes.io/projected/71ee0cd9-3e1e-43b8-8607-f3d827149245-kube-api-access-6lkwx\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:08.780904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.780861 2575 generic.go:356] "Generic (PLEG): container finished" podID="71ee0cd9-3e1e-43b8-8607-f3d827149245" containerID="4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b" exitCode=0 Jul 26 15:36:08.781096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.780912 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" Jul 26 15:36:08.781096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.780943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" event={"ID":"71ee0cd9-3e1e-43b8-8607-f3d827149245","Type":"ContainerDied","Data":"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b"} Jul 26 15:36:08.781096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.780996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm" event={"ID":"71ee0cd9-3e1e-43b8-8607-f3d827149245","Type":"ContainerDied","Data":"358bf02a0a3d5df817267caac3119d515f16330963641b728f49733aa549dd69"} Jul 26 15:36:08.781096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.781019 2575 scope.go:117] "RemoveContainer" containerID="4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b" Jul 26 15:36:08.790651 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.790632 2575 scope.go:117] "RemoveContainer" containerID="4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b" Jul 26 15:36:08.790932 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:36:08.790915 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b\": container with ID starting with 4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b not found: ID does not exist" containerID="4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b" Jul 26 15:36:08.790978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.790941 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b"} err="failed to get container status \"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b\": rpc error: code = NotFound desc = could not find container \"4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b\": container with ID starting with 4a0c3522cd45fde954e58a248e51c477323cc832a85e8c78d5619f724ec84a8b not found: ID does not exist" Jul 26 15:36:08.802822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.802768 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:36:08.806079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:08.806047 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-50ce075b-64d5d868d-wflsm"] Jul 26 15:36:10.554039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:10.554005 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ee0cd9-3e1e-43b8-8607-f3d827149245" path="/var/lib/kubelet/pods/71ee0cd9-3e1e-43b8-8607-f3d827149245/volumes" Jul 26 15:36:11.029099 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:11.029050 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:36:12.543745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:12.543707 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2"] Jul 26 15:36:12.545811 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:12.545790 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50ce075b-k96g2"] Jul 26 15:36:12.553917 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:12.553890 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60e8084-ad4a-4a85-95be-ed47f6340746" path="/var/lib/kubelet/pods/b60e8084-ad4a-4a85-95be-ed47f6340746/volumes" Jul 26 15:36:12.676426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:12.676390 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:36:12.676899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:12.676864 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" containerID="cri-o://5eb02844bcf89c83acac4801a38f27553d906bedec3245dd727fbb134c0ce7b1" gracePeriod=30 Jul 26 15:36:17.452706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.452663 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:17.453137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453069 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60e8084-ad4a-4a85-95be-ed47f6340746" containerName="revoke-keys" Jul 26 15:36:17.453137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453083 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60e8084-ad4a-4a85-95be-ed47f6340746" containerName="revoke-keys" Jul 26 15:36:17.453137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453106 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ee0cd9-3e1e-43b8-8607-f3d827149245" containerName="payload-pre-processing" Jul 26 15:36:17.453137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453111 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ee0cd9-3e1e-43b8-8607-f3d827149245" containerName="payload-pre-processing" Jul 26 15:36:17.453313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453179 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ee0cd9-3e1e-43b8-8607-f3d827149245" containerName="payload-pre-processing" Jul 26 15:36:17.453313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.453190 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60e8084-ad4a-4a85-95be-ed47f6340746" containerName="revoke-keys" Jul 26 15:36:17.457810 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.457784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.460688 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.460638 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-701ef13d-dockercfg-995ks\"" Jul 26 15:36:17.460978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.460959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-701ef13d\"" Jul 26 15:36:17.464786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.464754 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:17.470497 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.470474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:17.470607 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.470581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.481401 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.481373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:17.516127 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.516083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4778\" (UniqueName: \"kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.516331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.516159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.516331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.516200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.516331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.516216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xznm\" (UniqueName: \"kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.617713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.617672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4778\" (UniqueName: \"kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.617713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.617722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.617983 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.617850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.617983 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.617883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xznm\" (UniqueName: \"kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.618508 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.618489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.618621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.618603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.627177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.627147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xznm\" (UniqueName: \"kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm\") pod \"payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.627309 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.627229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4778\" (UniqueName: \"kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778\") pod \"payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.770822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.770716 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:17.783742 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.783707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:17.838779 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.838740 2575 generic.go:356] "Generic (PLEG): container finished" podID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerID="5eb02844bcf89c83acac4801a38f27553d906bedec3245dd727fbb134c0ce7b1" exitCode=0 Jul 26 15:36:17.838967 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.838952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerDied","Data":"5eb02844bcf89c83acac4801a38f27553d906bedec3245dd727fbb134c0ce7b1"} Jul 26 15:36:17.839044 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.839013 2575 scope.go:117] "RemoveContainer" containerID="d8c37d17ece91e2f63ee1255ebcbfa9508b497ac27e1f638cb9a6a7fa409d592" Jul 26 15:36:17.957463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.957435 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:17.960932 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.960908 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:36:17.962269 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:36:17.962238 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8ec948_df2b_4c0e_a4a6_6b648c11a467.slice/crio-9be2b98cd9a3474e3d80bf80734921c18a228ccfe51454c1cabdf573c44c8fca WatchSource:0}: Error finding container 9be2b98cd9a3474e3d80bf80734921c18a228ccfe51454c1cabdf573c44c8fca: Status 404 returned error can't find the container with id 9be2b98cd9a3474e3d80bf80734921c18a228ccfe51454c1cabdf573c44c8fca Jul 26 15:36:17.985306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:17.985232 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:17.987752 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:36:17.987715 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3c02f8_bcb4_482b_968c_e276f3fd09e8.slice/crio-118de05a75350b788aad745675270a2e074dd622c898138f2d4d71c2c6d16520 WatchSource:0}: Error finding container 118de05a75350b788aad745675270a2e074dd622c898138f2d4d71c2c6d16520: Status 404 returned error can't find the container with id 118de05a75350b788aad745675270a2e074dd622c898138f2d4d71c2c6d16520 Jul 26 15:36:18.021569 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021539 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lgj8\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021735 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021602 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021735 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021735 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021681 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021735 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021713 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021780 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021800 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.021964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.021884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token\") pod \"faa39021-cd12-4fce-bed5-59b852bc4d0e\" (UID: \"faa39021-cd12-4fce-bed5-59b852bc4d0e\") " Jul 26 15:36:18.022328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.022196 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:36:18.023135 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.022544 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:36:18.023135 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.022720 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:36:18.023135 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.022876 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:18.023478 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.023455 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:36:18.024661 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.024633 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8" (OuterVolumeSpecName: "kube-api-access-7lgj8") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "kube-api-access-7lgj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:36:18.025206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.025185 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:36:18.025351 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.025335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:36:18.025417 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.025379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "faa39021-cd12-4fce-bed5-59b852bc4d0e" (UID: "faa39021-cd12-4fce-bed5-59b852bc4d0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:36:18.123178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123144 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123176 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123185 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123194 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123203 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lgj8\" (UniqueName: \"kubernetes.io/projected/faa39021-cd12-4fce-bed5-59b852bc4d0e-kube-api-access-7lgj8\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123211 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123219 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa39021-cd12-4fce-bed5-59b852bc4d0e-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123226 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa39021-cd12-4fce-bed5-59b852bc4d0e-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.123475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.123245 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa39021-cd12-4fce-bed5-59b852bc4d0e-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:18.846017 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.845929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" event={"ID":"8e3c02f8-bcb4-482b-968c-e276f3fd09e8","Type":"ContainerStarted","Data":"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df"} Jul 26 15:36:18.846017 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.845982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" event={"ID":"8e3c02f8-bcb4-482b-968c-e276f3fd09e8","Type":"ContainerStarted","Data":"118de05a75350b788aad745675270a2e074dd622c898138f2d4d71c2c6d16520"} Jul 26 15:36:18.846522 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.846052 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:18.847760 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.847737 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" event={"ID":"ec8ec948-df2b-4c0e-a4a6-6b648c11a467","Type":"ContainerStarted","Data":"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951"} Jul 26 15:36:18.847833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.847784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" event={"ID":"ec8ec948-df2b-4c0e-a4a6-6b648c11a467","Type":"ContainerStarted","Data":"9be2b98cd9a3474e3d80bf80734921c18a228ccfe51454c1cabdf573c44c8fca"} Jul 26 15:36:18.847833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.847814 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:18.848975 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.848955 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" Jul 26 15:36:18.849094 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.848987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl" event={"ID":"faa39021-cd12-4fce-bed5-59b852bc4d0e","Type":"ContainerDied","Data":"7b721fc5de852c76fa37241dc6cbd3598fe78842fd4da05365739e9c75088ba3"} Jul 26 15:36:18.849094 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.849026 2575 scope.go:117] "RemoveContainer" containerID="5eb02844bcf89c83acac4801a38f27553d906bedec3245dd727fbb134c0ce7b1" Jul 26 15:36:18.861888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.861830 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" podStartSLOduration=1.861815258 podStartE2EDuration="1.861815258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:36:18.861003021 +0000 UTC m=+1852.758678020" watchObservedRunningTime="2026-07-26 15:36:18.861815258 +0000 UTC m=+1852.759490302" Jul 26 15:36:18.881248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.881211 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:36:18.883985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.883956 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-50ce075b-openshift-default-5986f687d6-986fl"] Jul 26 15:36:18.896874 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:18.896828 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" podStartSLOduration=1.89681413 podStartE2EDuration="1.89681413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:36:18.895466063 +0000 UTC m=+1852.793141063" watchObservedRunningTime="2026-07-26 15:36:18.89681413 +0000 UTC m=+1852.794489127" Jul 26 15:36:20.554841 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:20.554803 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" path="/var/lib/kubelet/pods/faa39021-cd12-4fce-bed5-59b852bc4d0e/volumes" Jul 26 15:36:22.835486 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:22.835445 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:36:29.858000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:29.857968 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:29.858439 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:29.858132 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:41.674532 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674490 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9"] Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674878 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674889 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674908 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674914 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674974 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.675125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.674981 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa39021-cd12-4fce-bed5-59b852bc4d0e" containerName="istio-proxy" Jul 26 15:36:41.678129 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.678110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:41.680635 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.680613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d665x\"" Jul 26 15:36:41.685056 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.685028 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9"] Jul 26 15:36:41.842257 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.842216 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9\" (UID: \"6b459683-e18a-40cd-ae5b-555cd7668593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:41.943324 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.943252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9\" (UID: \"6b459683-e18a-40cd-ae5b-555cd7668593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:41.943944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.943915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9\" (UID: \"6b459683-e18a-40cd-ae5b-555cd7668593\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:41.990823 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:41.990788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:42.345475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:42.345448 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9"] Jul 26 15:36:42.346768 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:36:42.346728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b459683_e18a_40cd_ae5b_555cd7668593.slice/crio-6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9 WatchSource:0}: Error finding container 6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9: Status 404 returned error can't find the container with id 6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9 Jul 26 15:36:42.985349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:42.985236 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" event={"ID":"6b459683-e18a-40cd-ae5b-555cd7668593","Type":"ContainerStarted","Data":"df1b0bd0aa2ceaf7a6ab45295ff10048feb594697b58fc693ed7f277da78b0a4"} Jul 26 15:36:42.985349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:42.985318 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" event={"ID":"6b459683-e18a-40cd-ae5b-555cd7668593","Type":"ContainerStarted","Data":"6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9"} Jul 26 15:36:42.999872 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:42.999814 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" podStartSLOduration=1.99979879 podStartE2EDuration="1.99979879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:36:42.999071904 +0000 UTC m=+1876.896746902" watchObservedRunningTime="2026-07-26 15:36:42.99979879 +0000 UTC m=+1876.897473787" Jul 26 15:36:43.992191 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:43.992153 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b459683-e18a-40cd-ae5b-555cd7668593" containerID="df1b0bd0aa2ceaf7a6ab45295ff10048feb594697b58fc693ed7f277da78b0a4" exitCode=0 Jul 26 15:36:43.992605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:43.992247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" event={"ID":"6b459683-e18a-40cd-ae5b-555cd7668593","Type":"ContainerDied","Data":"df1b0bd0aa2ceaf7a6ab45295ff10048feb594697b58fc693ed7f277da78b0a4"} Jul 26 15:36:45.138809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:45.138784 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:45.272911 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:45.272806 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca\") pod \"6b459683-e18a-40cd-ae5b-555cd7668593\" (UID: \"6b459683-e18a-40cd-ae5b-555cd7668593\") " Jul 26 15:36:45.273197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:45.273173 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b459683-e18a-40cd-ae5b-555cd7668593" (UID: "6b459683-e18a-40cd-ae5b-555cd7668593"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:45.374462 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:45.374426 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b459683-e18a-40cd-ae5b-555cd7668593-maas-api-service-ca\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:46.005391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:46.005349 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" Jul 26 15:36:46.005550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:46.005351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9" event={"ID":"6b459683-e18a-40cd-ae5b-555cd7668593","Type":"ContainerDied","Data":"6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9"} Jul 26 15:36:46.005550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:46.005484 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb1c6bdc59e7d39c0ef7be0b4909734a93b2099a428b9e9a7b66c9775ecd2f9" Jul 26 15:36:52.299896 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.299806 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:52.300359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.300100 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" podUID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" containerName="payload-processing" containerID="cri-o://646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df" gracePeriod=30 Jul 26 15:36:52.308434 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.308396 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:52.309022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.308708 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" podUID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" containerName="payload-pre-processing" containerID="cri-o://21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951" gracePeriod=30 Jul 26 15:36:52.592069 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.592041 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:52.595136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.595109 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:52.742643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.742598 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume\") pod \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " Jul 26 15:36:52.742856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.742693 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xznm\" (UniqueName: \"kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm\") pod \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\" (UID: \"8e3c02f8-bcb4-482b-968c-e276f3fd09e8\") " Jul 26 15:36:52.742856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.742721 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4778\" (UniqueName: \"kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778\") pod \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " Jul 26 15:36:52.742856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.742784 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume\") pod \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\" (UID: \"ec8ec948-df2b-4c0e-a4a6-6b648c11a467\") " Jul 26 15:36:52.743074 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.743042 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e3c02f8-bcb4-482b-968c-e276f3fd09e8" (UID: "8e3c02f8-bcb4-482b-968c-e276f3fd09e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:52.743218 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.743194 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec8ec948-df2b-4c0e-a4a6-6b648c11a467" (UID: "ec8ec948-df2b-4c0e-a4a6-6b648c11a467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:36:52.744994 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.744969 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm" (OuterVolumeSpecName: "kube-api-access-6xznm") pod "8e3c02f8-bcb4-482b-968c-e276f3fd09e8" (UID: "8e3c02f8-bcb4-482b-968c-e276f3fd09e8"). InnerVolumeSpecName "kube-api-access-6xznm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:36:52.745084 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.745017 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778" (OuterVolumeSpecName: "kube-api-access-w4778") pod "ec8ec948-df2b-4c0e-a4a6-6b648c11a467" (UID: "ec8ec948-df2b-4c0e-a4a6-6b648c11a467"). InnerVolumeSpecName "kube-api-access-w4778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:36:52.843780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.843682 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xznm\" (UniqueName: \"kubernetes.io/projected/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-kube-api-access-6xznm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:52.843780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.843719 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4778\" (UniqueName: \"kubernetes.io/projected/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-kube-api-access-w4778\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:52.843780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.843728 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8ec948-df2b-4c0e-a4a6-6b648c11a467-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:52.843780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:52.843738 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3c02f8-bcb4-482b-968c-e276f3fd09e8-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:36:53.048129 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.048092 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" containerID="646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df" exitCode=0 Jul 26 15:36:53.048333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.048138 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" Jul 26 15:36:53.048333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.048181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" event={"ID":"8e3c02f8-bcb4-482b-968c-e276f3fd09e8","Type":"ContainerDied","Data":"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df"} Jul 26 15:36:53.048333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.048245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q" event={"ID":"8e3c02f8-bcb4-482b-968c-e276f3fd09e8","Type":"ContainerDied","Data":"118de05a75350b788aad745675270a2e074dd622c898138f2d4d71c2c6d16520"} Jul 26 15:36:53.048333 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.048299 2575 scope.go:117] "RemoveContainer" containerID="646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df" Jul 26 15:36:53.049596 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.049572 2575 generic.go:356] "Generic (PLEG): container finished" podID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" containerID="21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951" exitCode=0 Jul 26 15:36:53.049694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.049600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" event={"ID":"ec8ec948-df2b-4c0e-a4a6-6b648c11a467","Type":"ContainerDied","Data":"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951"} Jul 26 15:36:53.049694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.049615 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" Jul 26 15:36:53.049694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.049650 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr" event={"ID":"ec8ec948-df2b-4c0e-a4a6-6b648c11a467","Type":"ContainerDied","Data":"9be2b98cd9a3474e3d80bf80734921c18a228ccfe51454c1cabdf573c44c8fca"} Jul 26 15:36:53.058193 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.058169 2575 scope.go:117] "RemoveContainer" containerID="646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df" Jul 26 15:36:53.058552 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:36:53.058514 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df\": container with ID starting with 646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df not found: ID does not exist" containerID="646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df" Jul 26 15:36:53.058640 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.058555 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df"} err="failed to get container status \"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df\": rpc error: code = NotFound desc = could not find container \"646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df\": container with ID starting with 646a6c6657d6cc1fc8ad33b379884c895d9ae05c81579be58236fd8b997bd4df not found: ID does not exist" Jul 26 15:36:53.058640 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.058578 2575 scope.go:117] "RemoveContainer" containerID="21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951" Jul 26 15:36:53.067937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.067856 2575 scope.go:117] "RemoveContainer" containerID="21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951" Jul 26 15:36:53.068266 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:36:53.068204 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951\": container with ID starting with 21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951 not found: ID does not exist" containerID="21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951" Jul 26 15:36:53.068266 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.068236 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951"} err="failed to get container status \"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951\": rpc error: code = NotFound desc = could not find container \"21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951\": container with ID starting with 21e675b49c6ecf94d06d00c5827eaa037ad45ce878d90c06e62c43daa94c6951 not found: ID does not exist" Jul 26 15:36:53.073772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.073747 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:53.080961 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.080937 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-701ef13d-d7b5ffc69-f986q"] Jul 26 15:36:53.092213 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.092188 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:53.098560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:53.098496 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-701ef13d-7fbb9f6964-222rr"] Jul 26 15:36:54.559736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:54.559704 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" path="/var/lib/kubelet/pods/8e3c02f8-bcb4-482b-968c-e276f3fd09e8/volumes" Jul 26 15:36:54.560124 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:54.560058 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" path="/var/lib/kubelet/pods/ec8ec948-df2b-4c0e-a4a6-6b648c11a467/volumes" Jul 26 15:36:55.324675 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:55.324630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:36:56.752391 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:56.752350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9"] Jul 26 15:36:56.754429 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:56.754406 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-701ef13d-b9hq9"] Jul 26 15:36:57.669984 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.669951 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:36:57.670362 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670348 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" containerName="payload-processing" Jul 26 15:36:57.670415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670363 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" containerName="payload-processing" Jul 26 15:36:57.670415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670384 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b459683-e18a-40cd-ae5b-555cd7668593" containerName="revoke-keys" Jul 26 15:36:57.670415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670389 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b459683-e18a-40cd-ae5b-555cd7668593" containerName="revoke-keys" Jul 26 15:36:57.670415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670397 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" containerName="payload-pre-processing" Jul 26 15:36:57.670415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670402 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" containerName="payload-pre-processing" Jul 26 15:36:57.670571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670456 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3c02f8-bcb4-482b-968c-e276f3fd09e8" containerName="payload-processing" Jul 26 15:36:57.670571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670463 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8ec948-df2b-4c0e-a4a6-6b648c11a467" containerName="payload-pre-processing" Jul 26 15:36:57.670571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.670471 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b459683-e18a-40cd-ae5b-555cd7668593" containerName="revoke-keys" Jul 26 15:36:57.673471 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.673456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.676055 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.676031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-340f592c-openshift-default-dockercfg-q597x\"" Jul 26 15:36:57.681060 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.681376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.681370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdrg\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.686144 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.686120 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:36:57.782516 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdrg\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.782958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.783422 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.782981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.783422 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.783004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.783422 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.783196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.783572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.783542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.783794 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.783778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.785528 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.785506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.785915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.785892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.791127 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.791101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.791631 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.791609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdrg\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg\") pod \"e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:57.986463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:57.986422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:58.131838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:58.131809 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:36:58.133586 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:36:58.133559 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535f8411_0d36_4026_8448_0069341ceca1.slice/crio-e4f97fdbda300dbff698c63db068947726c5324b56e2b5fe1adec8c880f001e0 WatchSource:0}: Error finding container e4f97fdbda300dbff698c63db068947726c5324b56e2b5fe1adec8c880f001e0: Status 404 returned error can't find the container with id e4f97fdbda300dbff698c63db068947726c5324b56e2b5fe1adec8c880f001e0 Jul 26 15:36:58.136068 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:58.136032 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:36:58.136155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:58.136101 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:36:58.136155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:58.136126 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:36:58.554741 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:58.554706 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b459683-e18a-40cd-ae5b-555cd7668593" path="/var/lib/kubelet/pods/6b459683-e18a-40cd-ae5b-555cd7668593/volumes" Jul 26 15:36:59.091426 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:59.091390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerStarted","Data":"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3"} Jul 26 15:36:59.091829 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:59.091442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerStarted","Data":"e4f97fdbda300dbff698c63db068947726c5324b56e2b5fe1adec8c880f001e0"} Jul 26 15:36:59.110952 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:59.110879 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podStartSLOduration=2.110849473 podStartE2EDuration="2.110849473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:36:59.10973789 +0000 UTC m=+1893.007412888" watchObservedRunningTime="2026-07-26 15:36:59.110849473 +0000 UTC m=+1893.008524471" Jul 26 15:36:59.987556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:59.987510 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:36:59.992651 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:36:59.992621 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:00.097327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:00.097291 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:00.098257 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:00.098235 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:01.140244 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.140205 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:01.143944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.143927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.146449 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.146429 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-340f592c\"" Jul 26 15:37:01.146920 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.146897 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-340f592c-dockercfg-7qm9f\"" Jul 26 15:37:01.155537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.155510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:01.161147 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.161124 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:01.165205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.165180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.179857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.179818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:01.216725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.216679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svprq\" (UniqueName: \"kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.216931 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.216895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghz4c\" (UniqueName: \"kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.217012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.216934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.217012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.216982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.318055 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghz4c\" (UniqueName: \"kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.318260 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.318260 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.318260 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svprq\" (UniqueName: \"kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.318955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.318955 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.318948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.326847 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.326813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svprq\" (UniqueName: \"kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq\") pod \"payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.326989 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.326863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghz4c\" (UniqueName: \"kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c\") pod \"payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.455286 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.455248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:01.478557 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.478527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:01.617731 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.617629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:01.621136 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:37:01.621088 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd3aac5_d903_4762_827c_f4fd27cd1980.slice/crio-4fd51d3ac25bb6a1a1208140e585c40b5aa2b5fddf34d66a994359a016c69c80 WatchSource:0}: Error finding container 4fd51d3ac25bb6a1a1208140e585c40b5aa2b5fddf34d66a994359a016c69c80: Status 404 returned error can't find the container with id 4fd51d3ac25bb6a1a1208140e585c40b5aa2b5fddf34d66a994359a016c69c80 Jul 26 15:37:01.656305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:01.656244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:01.658657 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:37:01.658611 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72fb30f_28b7_4b14_bc64_3349ca71ae93.slice/crio-7b8b273d7e6ed744a8a8009251259bd3751b82f0ac373aca8e84bf5bd017e1a8 WatchSource:0}: Error finding container 7b8b273d7e6ed744a8a8009251259bd3751b82f0ac373aca8e84bf5bd017e1a8: Status 404 returned error can't find the container with id 7b8b273d7e6ed744a8a8009251259bd3751b82f0ac373aca8e84bf5bd017e1a8 Jul 26 15:37:02.109316 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.109260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" event={"ID":"e72fb30f-28b7-4b14-bc64-3349ca71ae93","Type":"ContainerStarted","Data":"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b"} Jul 26 15:37:02.109523 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.109425 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:02.109523 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.109445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" event={"ID":"e72fb30f-28b7-4b14-bc64-3349ca71ae93","Type":"ContainerStarted","Data":"7b8b273d7e6ed744a8a8009251259bd3751b82f0ac373aca8e84bf5bd017e1a8"} Jul 26 15:37:02.111151 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.111126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" event={"ID":"7bd3aac5-d903-4762-827c-f4fd27cd1980","Type":"ContainerStarted","Data":"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6"} Jul 26 15:37:02.111259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.111163 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" event={"ID":"7bd3aac5-d903-4762-827c-f4fd27cd1980","Type":"ContainerStarted","Data":"4fd51d3ac25bb6a1a1208140e585c40b5aa2b5fddf34d66a994359a016c69c80"} Jul 26 15:37:02.111259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.111184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:02.126922 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.126855 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" podStartSLOduration=1.126838575 podStartE2EDuration="1.126838575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:37:02.124480631 +0000 UTC m=+1896.022155628" watchObservedRunningTime="2026-07-26 15:37:02.126838575 +0000 UTC m=+1896.024513605" Jul 26 15:37:02.140786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:02.140735 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" podStartSLOduration=1.140721044 podStartE2EDuration="1.140721044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:37:02.139586845 +0000 UTC m=+1896.037261844" watchObservedRunningTime="2026-07-26 15:37:02.140721044 +0000 UTC m=+1896.038396040" Jul 26 15:37:06.639210 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:06.639170 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:37:13.120558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:13.120517 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:13.121165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:13.120610 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:15.099458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.099417 2575 patch_prober.go:28] interesting pod/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:37:15.099954 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.099496 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 26 15:37:15.193012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.192978 2575 generic.go:356] "Generic (PLEG): container finished" podID="535f8411-0d36-4026-8448-0069341ceca1" containerID="603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3" exitCode=0 Jul 26 15:37:15.193178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.193048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerDied","Data":"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3"} Jul 26 15:37:15.193469 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.193451 2575 scope.go:117] "RemoveContainer" containerID="603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3" Jul 26 15:37:15.194471 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.194425 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:15.194533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.194515 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:15.194571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:15.194550 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:16.205261 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:16.205221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerStarted","Data":"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd"} Jul 26 15:37:16.987495 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:16.987455 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:16.987725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:16.987590 2575 patch_prober.go:28] interesting pod/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:37:16.987725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:16.987634 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 26 15:37:17.987731 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:17.987689 2575 patch_prober.go:28] interesting pod/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:37:17.988407 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:17.988371 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 26 15:37:18.987628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:18.987584 2575 patch_prober.go:28] interesting pod/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:37:18.987934 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:18.987904 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 26 15:37:19.987904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:19.987863 2575 patch_prober.go:28] interesting pod/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:37:19.988091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:19.987920 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 26 15:37:20.991025 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:20.990992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:20.991428 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:20.991268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:20.992044 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:20.992025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:36.758501 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:36.758464 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:36.759035 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:36.758699 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" podUID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" containerName="payload-processing" containerID="cri-o://8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b" gracePeriod=30 Jul 26 15:37:36.765214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:36.765178 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:36.765546 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:36.765452 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" podUID="7bd3aac5-d903-4762-827c-f4fd27cd1980" containerName="payload-pre-processing" containerID="cri-o://1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6" gracePeriod=30 Jul 26 15:37:37.049650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.049623 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:37.053510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.053485 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:37.159536 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.159499 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghz4c\" (UniqueName: \"kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c\") pod \"7bd3aac5-d903-4762-827c-f4fd27cd1980\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " Jul 26 15:37:37.159536 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.159556 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume\") pod \"7bd3aac5-d903-4762-827c-f4fd27cd1980\" (UID: \"7bd3aac5-d903-4762-827c-f4fd27cd1980\") " Jul 26 15:37:37.159785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.159639 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svprq\" (UniqueName: \"kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq\") pod \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " Jul 26 15:37:37.159785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.159710 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume\") pod \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\" (UID: \"e72fb30f-28b7-4b14-bc64-3349ca71ae93\") " Jul 26 15:37:37.160096 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.160072 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bd3aac5-d903-4762-827c-f4fd27cd1980" (UID: "7bd3aac5-d903-4762-827c-f4fd27cd1980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:37:37.160204 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.160136 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e72fb30f-28b7-4b14-bc64-3349ca71ae93" (UID: "e72fb30f-28b7-4b14-bc64-3349ca71ae93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:37:37.161769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.161744 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c" (OuterVolumeSpecName: "kube-api-access-ghz4c") pod "7bd3aac5-d903-4762-827c-f4fd27cd1980" (UID: "7bd3aac5-d903-4762-827c-f4fd27cd1980"). InnerVolumeSpecName "kube-api-access-ghz4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:37:37.161867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.161811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq" (OuterVolumeSpecName: "kube-api-access-svprq") pod "e72fb30f-28b7-4b14-bc64-3349ca71ae93" (UID: "e72fb30f-28b7-4b14-bc64-3349ca71ae93"). InnerVolumeSpecName "kube-api-access-svprq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:37:37.260784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.260746 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghz4c\" (UniqueName: \"kubernetes.io/projected/7bd3aac5-d903-4762-827c-f4fd27cd1980-kube-api-access-ghz4c\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:37.260784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.260779 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd3aac5-d903-4762-827c-f4fd27cd1980-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:37.260784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.260788 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svprq\" (UniqueName: \"kubernetes.io/projected/e72fb30f-28b7-4b14-bc64-3349ca71ae93-kube-api-access-svprq\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:37.261032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.260797 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72fb30f-28b7-4b14-bc64-3349ca71ae93-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:37.322774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.322676 2575 generic.go:356] "Generic (PLEG): container finished" podID="7bd3aac5-d903-4762-827c-f4fd27cd1980" containerID="1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6" exitCode=0 Jul 26 15:37:37.322774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.322725 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" Jul 26 15:37:37.322972 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.322776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" event={"ID":"7bd3aac5-d903-4762-827c-f4fd27cd1980","Type":"ContainerDied","Data":"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6"} Jul 26 15:37:37.322972 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.322830 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9" event={"ID":"7bd3aac5-d903-4762-827c-f4fd27cd1980","Type":"ContainerDied","Data":"4fd51d3ac25bb6a1a1208140e585c40b5aa2b5fddf34d66a994359a016c69c80"} Jul 26 15:37:37.322972 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.322855 2575 scope.go:117] "RemoveContainer" containerID="1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6" Jul 26 15:37:37.324111 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.324089 2575 generic.go:356] "Generic (PLEG): container finished" podID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" containerID="8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b" exitCode=0 Jul 26 15:37:37.324216 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.324130 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" Jul 26 15:37:37.324216 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.324183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" event={"ID":"e72fb30f-28b7-4b14-bc64-3349ca71ae93","Type":"ContainerDied","Data":"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b"} Jul 26 15:37:37.324350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.324239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf" event={"ID":"e72fb30f-28b7-4b14-bc64-3349ca71ae93","Type":"ContainerDied","Data":"7b8b273d7e6ed744a8a8009251259bd3751b82f0ac373aca8e84bf5bd017e1a8"} Jul 26 15:37:37.332973 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.332944 2575 scope.go:117] "RemoveContainer" containerID="1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6" Jul 26 15:37:37.333330 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:37:37.333310 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6\": container with ID starting with 1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6 not found: ID does not exist" containerID="1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6" Jul 26 15:37:37.333411 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.333338 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6"} err="failed to get container status \"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6\": rpc error: code = NotFound desc = could not find container \"1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6\": container with ID starting with 1055a5f71cfc10cdabf99f8cb4ba80c01e20810d118ab1c30ed6eaffd95ccdc6 not found: ID does not exist" Jul 26 15:37:37.333411 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.333356 2575 scope.go:117] "RemoveContainer" containerID="8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b" Jul 26 15:37:37.341967 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.341950 2575 scope.go:117] "RemoveContainer" containerID="8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b" Jul 26 15:37:37.342322 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:37:37.342300 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b\": container with ID starting with 8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b not found: ID does not exist" containerID="8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b" Jul 26 15:37:37.342419 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.342329 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b"} err="failed to get container status \"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b\": rpc error: code = NotFound desc = could not find container \"8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b\": container with ID starting with 8c0d966c289a033538d1d052bee7db2fcc546681f69af7e825fb6114abd41c3b not found: ID does not exist" Jul 26 15:37:37.349526 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.349502 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:37.352548 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.352497 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-340f592c-5f6ff49cb8-7fxx9"] Jul 26 15:37:37.362464 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.362436 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:37.364547 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:37.364527 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-340f592c-7975cdb764-z8hrf"] Jul 26 15:37:38.554908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:38.554875 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd3aac5-d903-4762-827c-f4fd27cd1980" path="/var/lib/kubelet/pods/7bd3aac5-d903-4762-827c-f4fd27cd1980/volumes" Jul 26 15:37:38.555336 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:38.555321 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" path="/var/lib/kubelet/pods/e72fb30f-28b7-4b14-bc64-3349ca71ae93/volumes" Jul 26 15:37:40.031693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:40.031602 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:37:42.506444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:42.506399 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:37:42.506947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:42.506617 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" containerID="cri-o://1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd" gracePeriod=30 Jul 26 15:37:43.281349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281311 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:37:43.281852 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281835 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" containerName="payload-processing" Jul 26 15:37:43.281925 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281855 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" containerName="payload-processing" Jul 26 15:37:43.281925 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd3aac5-d903-4762-827c-f4fd27cd1980" containerName="payload-pre-processing" Jul 26 15:37:43.281925 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281888 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd3aac5-d903-4762-827c-f4fd27cd1980" containerName="payload-pre-processing" Jul 26 15:37:43.282071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.281994 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd3aac5-d903-4762-827c-f4fd27cd1980" containerName="payload-pre-processing" Jul 26 15:37:43.282071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.282009 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72fb30f-28b7-4b14-bc64-3349ca71ae93" containerName="payload-processing" Jul 26 15:37:43.286780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.286749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.289846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.289821 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ae91a44a-openshift-default-dockercfg-m54j8\"" Jul 26 15:37:43.300186 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.300152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:37:43.317694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317694 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzv2q\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.317905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.318120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.317930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzv2q\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.418873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419162 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.418920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.419554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419705 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.419598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419705 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.419617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419705 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.419674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.419817 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.419773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.421537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.421514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.421733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.421717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.427475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.427448 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.427633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.427614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzv2q\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q\") pod \"e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.598165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.598062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:43.749958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.749929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:37:43.751889 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:37:43.751864 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afefa14_162f_48b3_8c37_7e1c4b58feb0.slice/crio-29ecc833248b6fbeebfb2b45f595eeac7f35974cf788b1ae5329d546f96bf956 WatchSource:0}: Error finding container 29ecc833248b6fbeebfb2b45f595eeac7f35974cf788b1ae5329d546f96bf956: Status 404 returned error can't find the container with id 29ecc833248b6fbeebfb2b45f595eeac7f35974cf788b1ae5329d546f96bf956 Jul 26 15:37:43.753838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.753807 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:43.753921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.753870 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:43.753921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:43.753896 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:37:44.365985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:44.365945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerStarted","Data":"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6"} Jul 26 15:37:44.366618 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:44.366001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerStarted","Data":"29ecc833248b6fbeebfb2b45f595eeac7f35974cf788b1ae5329d546f96bf956"} Jul 26 15:37:44.387145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:44.387088 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podStartSLOduration=1.387068581 podStartE2EDuration="1.387068581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:37:44.38502799 +0000 UTC m=+1938.282702988" watchObservedRunningTime="2026-07-26 15:37:44.387068581 +0000 UTC m=+1938.284743581" Jul 26 15:37:44.598522 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:44.598485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:44.603592 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:44.603562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:45.371442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:45.371394 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:45.372383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:45.372361 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:37:46.519248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.519208 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:37:46.522969 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.522943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.525715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.525692 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ae91a44a-dockercfg-269wj\"" Jul 26 15:37:46.525843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.525712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ae91a44a\"" Jul 26 15:37:46.530971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.530946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:37:46.534628 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.534604 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:37:46.538046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.538026 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.552110 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.552080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.552378 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.552351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrjt\" (UniqueName: \"kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.555996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.555974 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:37:46.653675 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.653641 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgnv\" (UniqueName: \"kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.653846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.653761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.653846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.653810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.653944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.653843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrjt\" (UniqueName: \"kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.654410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.654392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.661956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.661925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrjt\" (UniqueName: \"kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt\") pod \"payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.755270 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.755225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgnv\" (UniqueName: \"kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.755473 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.755339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.756016 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.755996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.762879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.762848 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgnv\" (UniqueName: \"kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv\") pod \"payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.836688 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.836593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:46.851445 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.851412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:46.995491 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:46.995453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:37:46.997887 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:37:46.997848 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e891545_af57_4bc1_a6dd_6c9f33c2111e.slice/crio-146c330c20c302e933e6c10dd42d34d14a5ddf7e4e8a838338ecd19b0e191c65 WatchSource:0}: Error finding container 146c330c20c302e933e6c10dd42d34d14a5ddf7e4e8a838338ecd19b0e191c65: Status 404 returned error can't find the container with id 146c330c20c302e933e6c10dd42d34d14a5ddf7e4e8a838338ecd19b0e191c65 Jul 26 15:37:47.028707 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.028681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:37:47.030228 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:37:47.030206 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84475ed4_a46a_4104_a1f6_99afa39f44f7.slice/crio-caa6af7d5066a72ab89a5da4caf2767c5a4411d2cecd5ce86d1e0fce99162db2 WatchSource:0}: Error finding container caa6af7d5066a72ab89a5da4caf2767c5a4411d2cecd5ce86d1e0fce99162db2: Status 404 returned error can't find the container with id caa6af7d5066a72ab89a5da4caf2767c5a4411d2cecd5ce86d1e0fce99162db2 Jul 26 15:37:47.388594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.388486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" event={"ID":"1e891545-af57-4bc1-a6dd-6c9f33c2111e","Type":"ContainerStarted","Data":"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a"} Jul 26 15:37:47.388594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.388565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" event={"ID":"1e891545-af57-4bc1-a6dd-6c9f33c2111e","Type":"ContainerStarted","Data":"146c330c20c302e933e6c10dd42d34d14a5ddf7e4e8a838338ecd19b0e191c65"} Jul 26 15:37:47.388814 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.388620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:47.390382 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.390344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" event={"ID":"84475ed4-a46a-4104-a1f6-99afa39f44f7","Type":"ContainerStarted","Data":"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181"} Jul 26 15:37:47.390542 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.390396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" event={"ID":"84475ed4-a46a-4104-a1f6-99afa39f44f7","Type":"ContainerStarted","Data":"caa6af7d5066a72ab89a5da4caf2767c5a4411d2cecd5ce86d1e0fce99162db2"} Jul 26 15:37:47.390610 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.390557 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:37:47.418410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.418350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" podStartSLOduration=1.418332472 podStartE2EDuration="1.418332472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:37:47.417759196 +0000 UTC m=+1941.315434191" watchObservedRunningTime="2026-07-26 15:37:47.418332472 +0000 UTC m=+1941.316007469" Jul 26 15:37:47.442067 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.442009 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" podStartSLOduration=1.44199045 podStartE2EDuration="1.44199045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:37:47.440040779 +0000 UTC m=+1941.337715798" watchObservedRunningTime="2026-07-26 15:37:47.44199045 +0000 UTC m=+1941.339665446" Jul 26 15:37:47.769751 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.769716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:47.865762 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865727 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.865933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865797 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.865933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.865933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865840 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.865933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865874 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.865933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.865972 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866000 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866052 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szdrg\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg\") pod \"535f8411-0d36-4026-8448-0069341ceca1\" (UID: \"535f8411-0d36-4026-8448-0069341ceca1\") " Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866131 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:37:47.866348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:37:47.866784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866447 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:37:47.866784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866457 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.866784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866477 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.866784 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866491 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.866979 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.866956 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data" (OuterVolumeSpecName: "istio-data") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:37:47.868328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.868294 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token" (OuterVolumeSpecName: "istio-token") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:37:47.868461 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.868380 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:37:47.868461 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.868404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:37:47.868461 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.868410 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg" (OuterVolumeSpecName: "kube-api-access-szdrg") pod "535f8411-0d36-4026-8448-0069341ceca1" (UID: "535f8411-0d36-4026-8448-0069341ceca1"). InnerVolumeSpecName "kube-api-access-szdrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:37:47.967700 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967658 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/535f8411-0d36-4026-8448-0069341ceca1-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.967700 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967691 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.967700 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967700 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.967700 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967708 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/535f8411-0d36-4026-8448-0069341ceca1-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.967980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967716 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/535f8411-0d36-4026-8448-0069341ceca1-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:47.967980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:47.967724 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdrg\" (UniqueName: \"kubernetes.io/projected/535f8411-0d36-4026-8448-0069341ceca1-kube-api-access-szdrg\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:37:48.399080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.398976 2575 generic.go:356] "Generic (PLEG): container finished" podID="535f8411-0d36-4026-8448-0069341ceca1" containerID="1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd" exitCode=0 Jul 26 15:37:48.399080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.399070 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" Jul 26 15:37:48.399327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.399066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerDied","Data":"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd"} Jul 26 15:37:48.399327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.399205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4" event={"ID":"535f8411-0d36-4026-8448-0069341ceca1","Type":"ContainerDied","Data":"e4f97fdbda300dbff698c63db068947726c5324b56e2b5fe1adec8c880f001e0"} Jul 26 15:37:48.399327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.399233 2575 scope.go:117] "RemoveContainer" containerID="1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd" Jul 26 15:37:48.409950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.409928 2575 scope.go:117] "RemoveContainer" containerID="603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3" Jul 26 15:37:48.419330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.419024 2575 scope.go:117] "RemoveContainer" containerID="1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd" Jul 26 15:37:48.419475 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:37:48.419439 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd\": container with ID starting with 1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd not found: ID does not exist" containerID="1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd" Jul 26 15:37:48.419605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.419486 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd"} err="failed to get container status \"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd\": rpc error: code = NotFound desc = could not find container \"1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd\": container with ID starting with 1f76e6796e3b7d5f47ac876639ccbb4da70aee075356c19c0413ece61d79e5bd not found: ID does not exist" Jul 26 15:37:48.419605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.419512 2575 scope.go:117] "RemoveContainer" containerID="603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3" Jul 26 15:37:48.419859 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:37:48.419832 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3\": container with ID starting with 603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3 not found: ID does not exist" containerID="603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3" Jul 26 15:37:48.419916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.419871 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3"} err="failed to get container status \"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3\": rpc error: code = NotFound desc = could not find container \"603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3\": container with ID starting with 603a75a22a6522fc879346973c5d067e7c530b68fc83d5a0839b87deaaf6b8e3 not found: ID does not exist" Jul 26 15:37:48.421901 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.421882 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:37:48.426121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.426093 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-340f592c-openshift-default-75f4bdf548-mq2x4"] Jul 26 15:37:48.554873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:48.554840 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535f8411-0d36-4026-8448-0069341ceca1" path="/var/lib/kubelet/pods/535f8411-0d36-4026-8448-0069341ceca1/volumes" Jul 26 15:37:51.441366 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:51.441331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:37:58.401204 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:58.401170 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:37:58.401649 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:37:58.401243 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:38:00.373590 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.373549 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:00.374016 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.373621 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:00.478876 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.478837 2575 generic.go:356] "Generic (PLEG): container finished" podID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerID="1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6" exitCode=0 Jul 26 15:38:00.479078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.478907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerDied","Data":"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6"} Jul 26 15:38:00.479339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.479324 2575 scope.go:117] "RemoveContainer" containerID="1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6" Jul 26 15:38:00.480539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.480506 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:38:00.480647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.480591 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:38:00.480647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:00.480636 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:38:01.490185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:01.490128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerStarted","Data":"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e"} Jul 26 15:38:01.598430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:01.598383 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:01.598796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:01.598766 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:01.598993 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:01.598965 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:02.599531 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:02.599483 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:02.600034 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:02.599569 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:03.599378 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:03.599332 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:03.599617 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:03.599441 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:04.599091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:04.599047 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:04.599342 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:04.599123 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:05.598534 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:05.598491 2575 patch_prober.go:28] interesting pod/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:38:05.599058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:05.598560 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 15:38:06.602240 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:06.602209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:06.602732 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:06.602493 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:06.603238 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:06.603221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:21.177359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.177250 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:38:21.178151 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.178112 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" podUID="84475ed4-a46a-4104-a1f6-99afa39f44f7" containerName="payload-processing" containerID="cri-o://89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181" gracePeriod=30 Jul 26 15:38:21.184076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.184041 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:38:21.184563 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.184510 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" podUID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" containerName="payload-pre-processing" containerID="cri-o://7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a" gracePeriod=30 Jul 26 15:38:21.476720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.476694 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:38:21.479699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.479671 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:38:21.483465 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.483439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jrjt\" (UniqueName: \"kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt\") pod \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " Jul 26 15:38:21.483593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.483488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume\") pod \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\" (UID: \"1e891545-af57-4bc1-a6dd-6c9f33c2111e\") " Jul 26 15:38:21.483593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.483548 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdgnv\" (UniqueName: \"kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv\") pod \"84475ed4-a46a-4104-a1f6-99afa39f44f7\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " Jul 26 15:38:21.483593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.483570 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume\") pod \"84475ed4-a46a-4104-a1f6-99afa39f44f7\" (UID: \"84475ed4-a46a-4104-a1f6-99afa39f44f7\") " Jul 26 15:38:21.483966 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.483932 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e891545-af57-4bc1-a6dd-6c9f33c2111e" (UID: "1e891545-af57-4bc1-a6dd-6c9f33c2111e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:38:21.484194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.484165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84475ed4-a46a-4104-a1f6-99afa39f44f7" (UID: "84475ed4-a46a-4104-a1f6-99afa39f44f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:38:21.485922 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.485888 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv" (OuterVolumeSpecName: "kube-api-access-sdgnv") pod "84475ed4-a46a-4104-a1f6-99afa39f44f7" (UID: "84475ed4-a46a-4104-a1f6-99afa39f44f7"). InnerVolumeSpecName "kube-api-access-sdgnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:38:21.486142 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.486121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt" (OuterVolumeSpecName: "kube-api-access-4jrjt") pod "1e891545-af57-4bc1-a6dd-6c9f33c2111e" (UID: "1e891545-af57-4bc1-a6dd-6c9f33c2111e"). InnerVolumeSpecName "kube-api-access-4jrjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:38:21.584656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.584611 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrjt\" (UniqueName: \"kubernetes.io/projected/1e891545-af57-4bc1-a6dd-6c9f33c2111e-kube-api-access-4jrjt\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:21.584656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.584643 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e891545-af57-4bc1-a6dd-6c9f33c2111e-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:21.584656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.584653 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdgnv\" (UniqueName: \"kubernetes.io/projected/84475ed4-a46a-4104-a1f6-99afa39f44f7-kube-api-access-sdgnv\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:21.584656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.584662 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84475ed4-a46a-4104-a1f6-99afa39f44f7-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:21.617123 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.617081 2575 generic.go:356] "Generic (PLEG): container finished" podID="84475ed4-a46a-4104-a1f6-99afa39f44f7" containerID="89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181" exitCode=0 Jul 26 15:38:21.617123 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.617123 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" Jul 26 15:38:21.617386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.617164 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" event={"ID":"84475ed4-a46a-4104-a1f6-99afa39f44f7","Type":"ContainerDied","Data":"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181"} Jul 26 15:38:21.617386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.617234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh" event={"ID":"84475ed4-a46a-4104-a1f6-99afa39f44f7","Type":"ContainerDied","Data":"caa6af7d5066a72ab89a5da4caf2767c5a4411d2cecd5ce86d1e0fce99162db2"} Jul 26 15:38:21.617386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.617257 2575 scope.go:117] "RemoveContainer" containerID="89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181" Jul 26 15:38:21.618883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.618861 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" containerID="7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a" exitCode=0 Jul 26 15:38:21.619000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.618902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" event={"ID":"1e891545-af57-4bc1-a6dd-6c9f33c2111e","Type":"ContainerDied","Data":"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a"} Jul 26 15:38:21.619000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.618942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" event={"ID":"1e891545-af57-4bc1-a6dd-6c9f33c2111e","Type":"ContainerDied","Data":"146c330c20c302e933e6c10dd42d34d14a5ddf7e4e8a838338ecd19b0e191c65"} Jul 26 15:38:21.619000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.618954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q" Jul 26 15:38:21.627889 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.627868 2575 scope.go:117] "RemoveContainer" containerID="89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181" Jul 26 15:38:21.628216 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:38:21.628182 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181\": container with ID starting with 89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181 not found: ID does not exist" containerID="89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181" Jul 26 15:38:21.628342 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.628219 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181"} err="failed to get container status \"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181\": rpc error: code = NotFound desc = could not find container \"89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181\": container with ID starting with 89497a2752587eef128c22ce5ed3cc45dca4c073b48ed4ed4a4f59b703b2a181 not found: ID does not exist" Jul 26 15:38:21.628342 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.628242 2575 scope.go:117] "RemoveContainer" containerID="7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a" Jul 26 15:38:21.639322 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.639301 2575 scope.go:117] "RemoveContainer" containerID="7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a" Jul 26 15:38:21.639598 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:38:21.639576 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a\": container with ID starting with 7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a not found: ID does not exist" containerID="7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a" Jul 26 15:38:21.639664 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.639606 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a"} err="failed to get container status \"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a\": rpc error: code = NotFound desc = could not find container \"7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a\": container with ID starting with 7c8c788ad06ba42b8dadbe6d0193d53414a1cfe789b7cb2e1ce72d0e9a65d73a not found: ID does not exist" Jul 26 15:38:21.643113 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.643088 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:38:21.648838 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.648813 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ae91a44a-55bb8d569d-9vzkh"] Jul 26 15:38:21.659541 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.659508 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:38:21.661645 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:21.661626 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ae91a44a-67566cfb-lks8q"] Jul 26 15:38:22.554457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:22.554429 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" path="/var/lib/kubelet/pods/1e891545-af57-4bc1-a6dd-6c9f33c2111e/volumes" Jul 26 15:38:22.554857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:22.554824 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84475ed4-a46a-4104-a1f6-99afa39f44f7" path="/var/lib/kubelet/pods/84475ed4-a46a-4104-a1f6-99afa39f44f7/volumes" Jul 26 15:38:23.234630 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:23.234592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:38:25.787695 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:25.787663 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:38:25.788170 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:25.787920 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" containerID="cri-o://93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e" gracePeriod=30 Jul 26 15:38:31.047538 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.047468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:31.072544 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072506 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.072736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072572 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.072736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.072736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.072736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072665 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.073022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072845 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:38:31.073022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072916 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzv2q\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.073022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072959 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:38:31.073022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.072995 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.073218 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073026 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.073218 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo\") pod \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\" (UID: \"7afefa14-162f-48b3-8c37-7e1c4b58feb0\") " Jul 26 15:38:31.073218 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073096 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:38:31.073430 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073409 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.073490 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073435 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.073490 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073449 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.073821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.073782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:38:31.074216 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.074175 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:38:31.075438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.075402 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q" (OuterVolumeSpecName: "kube-api-access-tzv2q") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "kube-api-access-tzv2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:38:31.075550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.075529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:38:31.075683 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.075651 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:38:31.076210 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.076128 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7afefa14-162f-48b3-8c37-7e1c4b58feb0" (UID: "7afefa14-162f-48b3-8c37-7e1c4b58feb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:38:31.174792 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174744 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.174792 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174789 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.174792 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174798 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.175036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174806 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzv2q\" (UniqueName: \"kubernetes.io/projected/7afefa14-162f-48b3-8c37-7e1c4b58feb0-kube-api-access-tzv2q\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.175036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174816 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.175036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.174826 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afefa14-162f-48b3-8c37-7e1c4b58feb0-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:38:31.676314 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.676250 2575 generic.go:356] "Generic (PLEG): container finished" podID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerID="93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e" exitCode=0 Jul 26 15:38:31.676314 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.676314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerDied","Data":"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e"} Jul 26 15:38:31.676571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.676339 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" Jul 26 15:38:31.676571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.676361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c" event={"ID":"7afefa14-162f-48b3-8c37-7e1c4b58feb0","Type":"ContainerDied","Data":"29ecc833248b6fbeebfb2b45f595eeac7f35974cf788b1ae5329d546f96bf956"} Jul 26 15:38:31.676571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.676384 2575 scope.go:117] "RemoveContainer" containerID="93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e" Jul 26 15:38:31.686089 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.686067 2575 scope.go:117] "RemoveContainer" containerID="1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6" Jul 26 15:38:31.695116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.695088 2575 scope.go:117] "RemoveContainer" containerID="93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e" Jul 26 15:38:31.695550 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:38:31.695510 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e\": container with ID starting with 93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e not found: ID does not exist" containerID="93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e" Jul 26 15:38:31.695664 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.695582 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e"} err="failed to get container status \"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e\": rpc error: code = NotFound desc = could not find container \"93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e\": container with ID starting with 93a991152d0af0a52092749da8b81bf4a790f3a5b3e30810c2f4887a5b52158e not found: ID does not exist" Jul 26 15:38:31.695764 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.695667 2575 scope.go:117] "RemoveContainer" containerID="1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6" Jul 26 15:38:31.696180 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:38:31.696156 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6\": container with ID starting with 1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6 not found: ID does not exist" containerID="1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6" Jul 26 15:38:31.696299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.696189 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6"} err="failed to get container status \"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6\": rpc error: code = NotFound desc = could not find container \"1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6\": container with ID starting with 1c3bf5f86aacac2fc315bbeb98e49e86dfce8149ebb147a5367d7f71867422b6 not found: ID does not exist" Jul 26 15:38:31.698561 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.698538 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:38:31.702048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:31.702024 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ae91a44a-openshift-default-7989974996-6hn4c"] Jul 26 15:38:32.554094 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:38:32.554062 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" path="/var/lib/kubelet/pods/7afefa14-162f-48b3-8c37-7e1c4b58feb0/volumes" Jul 26 15:39:50.033554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:39:50.033466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:39:59.634173 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:39:59.634141 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:40:02.812415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812380 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812744 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" containerName="payload-pre-processing" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812755 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" containerName="payload-pre-processing" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812776 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812781 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812789 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812794 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812800 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84475ed4-a46a-4104-a1f6-99afa39f44f7" containerName="payload-processing" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812805 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84475ed4-a46a-4104-a1f6-99afa39f44f7" containerName="payload-processing" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812819 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:02.812822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812824 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812834 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812839 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812890 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e891545-af57-4bc1-a6dd-6c9f33c2111e" containerName="payload-pre-processing" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812899 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812905 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812910 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afefa14-162f-48b3-8c37-7e1c4b58feb0" containerName="istio-proxy" Jul 26 15:40:02.813134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.812916 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84475ed4-a46a-4104-a1f6-99afa39f44f7" containerName="payload-processing" Jul 26 15:40:02.815905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.815889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.818447 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.818423 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1d5bf042-openshift-default-dockercfg-v8tz8\"" Jul 26 15:40:02.837231 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.837198 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:02.896309 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896309 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs7t\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.896809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.896672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs7t\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.997948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.997992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998443 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:02.998857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:02.998839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.000211 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.000180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.000745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.000714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.007023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.007000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs7t\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.007512 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.007489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token\") pod \"e2e-mt-1d5bf042-openshift-default-f84f78944-crhml\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.126606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.126507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:03.274520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.274492 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:03.276616 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:40:03.276584 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85d1d19_af34_42a7_a8b3_363ab68d3c35.slice/crio-8eb0ba516b1aa36542908189456afeddc4309f5c2f3b23a1aaa9e79c60012a81 WatchSource:0}: Error finding container 8eb0ba516b1aa36542908189456afeddc4309f5c2f3b23a1aaa9e79c60012a81: Status 404 returned error can't find the container with id 8eb0ba516b1aa36542908189456afeddc4309f5c2f3b23a1aaa9e79c60012a81 Jul 26 15:40:03.279018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.278981 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:03.279120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.279068 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:03.279183 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:03.279116 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:04.198798 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:04.198750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerStarted","Data":"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa"} Jul 26 15:40:04.199292 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:04.198819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerStarted","Data":"8eb0ba516b1aa36542908189456afeddc4309f5c2f3b23a1aaa9e79c60012a81"} Jul 26 15:40:04.220177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:04.220117 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podStartSLOduration=2.220099485 podStartE2EDuration="2.220099485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:40:04.219146756 +0000 UTC m=+2078.116821772" watchObservedRunningTime="2026-07-26 15:40:04.220099485 +0000 UTC m=+2078.117774481" Jul 26 15:40:05.127008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:05.126949 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:05.132020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:05.131993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:05.204677 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:05.204641 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:05.205638 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:05.205614 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:07.135195 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.135155 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:07.135680 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.135663 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="535f8411-0d36-4026-8448-0069341ceca1" containerName="istio-proxy" Jul 26 15:40:07.138720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.138701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.141815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.141761 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1d5bf042-dockercfg-dkfts\"" Jul 26 15:40:07.141815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.141805 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1d5bf042\"" Jul 26 15:40:07.148522 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.148496 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:07.240345 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.240307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frjp\" (UniqueName: \"kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.240518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.240405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.341637 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.341597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frjp\" (UniqueName: \"kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.341826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.341689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.342473 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.342447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.350789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.350766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frjp\" (UniqueName: \"kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp\") pod \"payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.450327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.450268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:07.594480 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:07.594452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:07.596465 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:40:07.596423 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9cfadf_804b_4269_9bf8_8ccd02c90c9b.slice/crio-6813c12e84f15e1d3046e66e646a1914c522e6741a17229a40b8118000d17f19 WatchSource:0}: Error finding container 6813c12e84f15e1d3046e66e646a1914c522e6741a17229a40b8118000d17f19: Status 404 returned error can't find the container with id 6813c12e84f15e1d3046e66e646a1914c522e6741a17229a40b8118000d17f19 Jul 26 15:40:08.224791 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:08.224755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" event={"ID":"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b","Type":"ContainerStarted","Data":"a9f41a60a515cb58e533a87c9fe830391b7b8e71c3c782033a12891d30a31a55"} Jul 26 15:40:08.225201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:08.224810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" event={"ID":"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b","Type":"ContainerStarted","Data":"6813c12e84f15e1d3046e66e646a1914c522e6741a17229a40b8118000d17f19"} Jul 26 15:40:08.225201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:08.224867 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:08.243456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:08.243402 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" podStartSLOduration=1.243385722 podStartE2EDuration="1.243385722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:40:08.241354559 +0000 UTC m=+2082.139029558" watchObservedRunningTime="2026-07-26 15:40:08.243385722 +0000 UTC m=+2082.141060718" Jul 26 15:40:12.143145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:12.143105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:40:19.234824 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:19.234790 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:20.505785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:20.504962 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:54048->10.134.0.59:15021: read: connection reset by peer" start-of-body= Jul 26 15:40:20.505785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:20.505042 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:54048->10.134.0.59:15021: read: connection reset by peer" Jul 26 15:40:21.300890 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.300853 2575 generic.go:356] "Generic (PLEG): container finished" podID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerID="fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa" exitCode=0 Jul 26 15:40:21.301078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.300926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerDied","Data":"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa"} Jul 26 15:40:21.301402 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.301386 2575 scope.go:117] "RemoveContainer" containerID="fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa" Jul 26 15:40:21.302474 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.302441 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:21.302555 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.302528 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:21.302593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:21.302577 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:22.309043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:22.309004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerStarted","Data":"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db"} Jul 26 15:40:23.126819 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:23.126784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:23.127043 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:23.127018 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 15:40:23.127111 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:23.127068 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 15:40:24.127807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:24.127770 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 15:40:24.128211 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:24.127826 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 15:40:25.127171 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:25.127128 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 15:40:25.127393 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:25.127186 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 15:40:26.127886 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:26.127848 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 26 15:40:26.128438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:26.127917 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 26 15:40:27.130900 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:27.130871 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:27.131307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:27.131113 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:27.131834 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:27.131814 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:28.181826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.181788 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:28.182343 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.182011 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" podUID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" containerName="payload-processing" containerID="cri-o://a9f41a60a515cb58e533a87c9fe830391b7b8e71c3c782033a12891d30a31a55" gracePeriod=30 Jul 26 15:40:28.355554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.355510 2575 generic.go:356] "Generic (PLEG): container finished" podID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" containerID="a9f41a60a515cb58e533a87c9fe830391b7b8e71c3c782033a12891d30a31a55" exitCode=0 Jul 26 15:40:28.355921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.355887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" event={"ID":"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b","Type":"ContainerDied","Data":"a9f41a60a515cb58e533a87c9fe830391b7b8e71c3c782033a12891d30a31a55"} Jul 26 15:40:28.445752 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.445688 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:28.547650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.547608 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7frjp\" (UniqueName: \"kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp\") pod \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " Jul 26 15:40:28.547853 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.547690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume\") pod \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\" (UID: \"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b\") " Jul 26 15:40:28.548119 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.548090 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" (UID: "aa9cfadf-804b-4269-9bf8-8ccd02c90c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:28.549922 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.549896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp" (OuterVolumeSpecName: "kube-api-access-7frjp") pod "aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" (UID: "aa9cfadf-804b-4269-9bf8-8ccd02c90c9b"). InnerVolumeSpecName "kube-api-access-7frjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:40:28.649399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.649358 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frjp\" (UniqueName: \"kubernetes.io/projected/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-kube-api-access-7frjp\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:28.649399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:28.649389 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:29.362786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:29.362746 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" Jul 26 15:40:29.362786 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:29.362773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn" event={"ID":"aa9cfadf-804b-4269-9bf8-8ccd02c90c9b","Type":"ContainerDied","Data":"6813c12e84f15e1d3046e66e646a1914c522e6741a17229a40b8118000d17f19"} Jul 26 15:40:29.363375 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:29.362832 2575 scope.go:117] "RemoveContainer" containerID="a9f41a60a515cb58e533a87c9fe830391b7b8e71c3c782033a12891d30a31a55" Jul 26 15:40:29.379386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:29.379355 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:29.381606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:29.381581 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1d5bf042-6fc8cb9b86-75qvn"] Jul 26 15:40:30.533898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:30.533868 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:40:30.538970 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:30.538946 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:40:30.553751 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:30.553703 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" path="/var/lib/kubelet/pods/aa9cfadf-804b-4269-9bf8-8ccd02c90c9b/volumes" Jul 26 15:40:31.331487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:31.331450 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:40:39.158556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:39.158510 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:39.158944 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:39.158825 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" containerID="cri-o://af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db" gracePeriod=30 Jul 26 15:40:42.134134 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:42.134099 2575 patch_prober.go:28] interesting pod/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:40:42.134551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:42.134159 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:40:44.420941 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.420874 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:44.449819 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.449787 2575 generic.go:356] "Generic (PLEG): container finished" podID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerID="af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db" exitCode=0 Jul 26 15:40:44.449980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.449834 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" Jul 26 15:40:44.449980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.449874 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerDied","Data":"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db"} Jul 26 15:40:44.449980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.449941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml" event={"ID":"c85d1d19-af34-42a7-a8b3-363ab68d3c35","Type":"ContainerDied","Data":"8eb0ba516b1aa36542908189456afeddc4309f5c2f3b23a1aaa9e79c60012a81"} Jul 26 15:40:44.449980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.449977 2575 scope.go:117] "RemoveContainer" containerID="af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db" Jul 26 15:40:44.459586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.459565 2575 scope.go:117] "RemoveContainer" containerID="fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa" Jul 26 15:40:44.468003 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.467982 2575 scope.go:117] "RemoveContainer" containerID="af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db" Jul 26 15:40:44.468260 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:40:44.468241 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db\": container with ID starting with af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db not found: ID does not exist" containerID="af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db" Jul 26 15:40:44.468404 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.468287 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db"} err="failed to get container status \"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db\": rpc error: code = NotFound desc = could not find container \"af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db\": container with ID starting with af1f5010a09b6a6700dc02840ba69724897275e2c2e6b809b7c80227cc17c9db not found: ID does not exist" Jul 26 15:40:44.468404 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.468312 2575 scope.go:117] "RemoveContainer" containerID="fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa" Jul 26 15:40:44.468616 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:40:44.468599 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa\": container with ID starting with fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa not found: ID does not exist" containerID="fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa" Jul 26 15:40:44.468663 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.468620 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa"} err="failed to get container status \"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa\": rpc error: code = NotFound desc = could not find container \"fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa\": container with ID starting with fcbcc2b33a7aff15c7b39203650c78edabff0bd8b9888c4f91c345d761af3caa not found: ID does not exist" Jul 26 15:40:44.489537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489537 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489540 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489583 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489603 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489631 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489654 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489725 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489707 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqs7t\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489730 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.489971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:40:44.489971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.489943 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:40:44.490139 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token\") pod \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\" (UID: \"c85d1d19-af34-42a7-a8b3-363ab68d3c35\") " Jul 26 15:40:44.490139 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490008 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:40:44.490371 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490349 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.490476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490371 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.490476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490386 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.490760 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490524 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data" (OuterVolumeSpecName: "istio-data") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:40:44.490760 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.490547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:40:44.491897 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.491875 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:40:44.492080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.492063 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token" (OuterVolumeSpecName: "istio-token") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:40:44.492401 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.492375 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:40:44.492401 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.492384 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t" (OuterVolumeSpecName: "kube-api-access-fqs7t") pod "c85d1d19-af34-42a7-a8b3-363ab68d3c35" (UID: "c85d1d19-af34-42a7-a8b3-363ab68d3c35"). InnerVolumeSpecName "kube-api-access-fqs7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:40:44.591052 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591017 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.591052 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591045 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.591052 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591054 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.591305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591063 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.591305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591071 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqs7t\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-kube-api-access-fqs7t\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.591305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.591080 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c85d1d19-af34-42a7-a8b3-363ab68d3c35-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:40:44.765980 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.765949 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:44.769617 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:44.769587 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1d5bf042-openshift-default-f84f78944-crhml"] Jul 26 15:40:46.555348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:46.555316 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" path="/var/lib/kubelet/pods/c85d1d19-af34-42a7-a8b3-363ab68d3c35/volumes" Jul 26 15:40:58.447824 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.447790 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448161 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448173 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448186 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448193 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448210 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" containerName="payload-processing" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448216 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" containerName="payload-processing" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448268 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9cfadf-804b-4269-9bf8-8ccd02c90c9b" containerName="payload-processing" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448299 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.448347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.448306 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85d1d19-af34-42a7-a8b3-363ab68d3c35" containerName="istio-proxy" Jul 26 15:40:58.452749 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.452729 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.455259 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.455239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-52d355e1-openshift-default-dockercfg-d6vm9\"" Jul 26 15:40:58.463917 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.463882 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:40:58.513482 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513482 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rz7\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.513923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.513831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.614887 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.614851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rz7\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615065 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.614898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615190 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615454 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615541 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615601 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615659 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.615911 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.616015 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.616015 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.615985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.616475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.616454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.617679 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.617654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.617903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.617883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.623032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.623000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rz7\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.623355 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.623335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token\") pod \"e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.771603 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.771514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:58.921214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.921190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:40:58.922884 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:40:58.922848 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef4ddff_e42f_4937_98dd_7f97d4ba5302.slice/crio-48c7bd2dd27d5d50f523b7885de9708076bf2fcd90de9931cd3aa89c89a9337f WatchSource:0}: Error finding container 48c7bd2dd27d5d50f523b7885de9708076bf2fcd90de9931cd3aa89c89a9337f: Status 404 returned error can't find the container with id 48c7bd2dd27d5d50f523b7885de9708076bf2fcd90de9931cd3aa89c89a9337f Jul 26 15:40:58.924871 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.924836 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:58.924951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.924913 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:58.924951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:58.924943 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:40:59.534744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:59.534703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerStarted","Data":"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784"} Jul 26 15:40:59.535165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:59.534769 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerStarted","Data":"48c7bd2dd27d5d50f523b7885de9708076bf2fcd90de9931cd3aa89c89a9337f"} Jul 26 15:40:59.555041 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:59.554988 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podStartSLOduration=1.5549736539999999 podStartE2EDuration="1.554973654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:40:59.551561264 +0000 UTC m=+2133.449236261" watchObservedRunningTime="2026-07-26 15:40:59.554973654 +0000 UTC m=+2133.452648651" Jul 26 15:40:59.772004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:59.771961 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:40:59.777481 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:40:59.777449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:00.541132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:00.541099 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:00.542046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:00.542026 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:02.202268 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.202231 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:41:02.205745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.205722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.208220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.208201 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-52d355e1-dockercfg-dv7m9\"" Jul 26 15:41:02.208405 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.208227 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-52d355e1\"" Jul 26 15:41:02.224020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.223989 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:41:02.351656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.351611 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr92\" (UniqueName: \"kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.351656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.351661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.453076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.452976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr92\" (UniqueName: \"kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.453076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.453028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.453747 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.453724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.462505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.462467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr92\" (UniqueName: \"kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92\") pod \"payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.516953 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.516915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:02.663099 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:02.663064 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:41:02.664492 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:41:02.664466 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10bf91b_1c01_4aa7_ae02_20c8a9d8b68f.slice/crio-9ef9a74b0990f93a90b956bba332ee6250c12be615a4838768743dbf79dee5ce WatchSource:0}: Error finding container 9ef9a74b0990f93a90b956bba332ee6250c12be615a4838768743dbf79dee5ce: Status 404 returned error can't find the container with id 9ef9a74b0990f93a90b956bba332ee6250c12be615a4838768743dbf79dee5ce Jul 26 15:41:03.559967 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:03.559915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" event={"ID":"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f","Type":"ContainerStarted","Data":"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442"} Jul 26 15:41:03.560456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:03.559982 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:03.560456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:03.560001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" event={"ID":"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f","Type":"ContainerStarted","Data":"9ef9a74b0990f93a90b956bba332ee6250c12be615a4838768743dbf79dee5ce"} Jul 26 15:41:03.578411 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:03.578356 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" podStartSLOduration=1.578336333 podStartE2EDuration="1.578336333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:41:03.575734423 +0000 UTC m=+2137.473409441" watchObservedRunningTime="2026-07-26 15:41:03.578336333 +0000 UTC m=+2137.476011330" Jul 26 15:41:08.034968 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:08.034926 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:41:14.572206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:14.572170 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:41:15.935564 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:15.935026 2575 patch_prober.go:28] interesting pod/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:42660->10.134.0.61:15021: read: connection reset by peer" start-of-body= Jul 26 15:41:15.935564 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:15.935100 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": read tcp 10.134.0.2:42660->10.134.0.61:15021: read: connection reset by peer" Jul 26 15:41:16.644352 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.644318 2575 generic.go:356] "Generic (PLEG): container finished" podID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerID="1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784" exitCode=0 Jul 26 15:41:16.644546 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.644401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerDied","Data":"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784"} Jul 26 15:41:16.644853 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.644836 2575 scope.go:117] "RemoveContainer" containerID="1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784" Jul 26 15:41:16.645897 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.645864 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:41:16.645993 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.645958 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:41:16.646058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:16.646009 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:41:17.651759 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:17.651720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerStarted","Data":"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54"} Jul 26 15:41:17.772720 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:17.772684 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:17.772907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:17.772884 2575 patch_prober.go:28] interesting pod/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 15:41:17.772962 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:17.772944 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 15:41:18.772929 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:18.772882 2575 patch_prober.go:28] interesting pod/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 15:41:18.773365 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:18.772949 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 15:41:19.772393 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:19.772355 2575 patch_prober.go:28] interesting pod/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 15:41:19.772580 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:19.772412 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 15:41:20.771961 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:20.771918 2575 patch_prober.go:28] interesting pod/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 26 15:41:20.772377 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:20.771991 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 26 15:41:21.775968 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:21.775930 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:21.776440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:21.776366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:21.777205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:21.777184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:41:25.434387 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.434331 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:41:25.438334 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.438315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.442125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.442099 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-52d355e1\"/\"default-dockercfg-qzfxf\"" Jul 26 15:41:25.442269 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.442158 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-52d355e1\"/\"e2e-lifecycle-model-52d355e1-kserve-self-signed-certs\"" Jul 26 15:41:25.442269 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.442231 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-52d355e1\"/\"kube-root-ca.crt\"" Jul 26 15:41:25.442269 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.442234 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-52d355e1\"/\"openshift-service-ca.crt\"" Jul 26 15:41:25.447756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.447731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:41:25.475149 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.475348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf78m\" (UniqueName: \"kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.475348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.475348 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.475487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.475487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.475411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.575907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.575867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.575907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.575920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf78m\" (UniqueName: \"kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576192 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.575964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576192 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576192 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576192 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576480 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.576824 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.576782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.578795 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.578767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.578904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.578885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.585685 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.585658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf78m\" (UniqueName: \"kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m\") pod \"e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.748225 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.748130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:25.893872 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:25.893844 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:41:25.894270 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:41:25.894240 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae2052ff_c656_4804_891a_8c17ca56f0eb.slice/crio-61a9befe4ee1e2afd5506453c839122acf8963b293753496c0eeec6ae4e76091 WatchSource:0}: Error finding container 61a9befe4ee1e2afd5506453c839122acf8963b293753496c0eeec6ae4e76091: Status 404 returned error can't find the container with id 61a9befe4ee1e2afd5506453c839122acf8963b293753496c0eeec6ae4e76091 Jul 26 15:41:26.708349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:26.708311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" event={"ID":"ae2052ff-c656-4804-891a-8c17ca56f0eb","Type":"ContainerStarted","Data":"cdb0448ce240447fe726e22af9042d8a8fc0449b1dc754807ae509439f76e94a"} Jul 26 15:41:26.708788 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:26.708361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" event={"ID":"ae2052ff-c656-4804-891a-8c17ca56f0eb","Type":"ContainerStarted","Data":"61a9befe4ee1e2afd5506453c839122acf8963b293753496c0eeec6ae4e76091"} Jul 26 15:41:26.726030 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:26.725980 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" podStartSLOduration=1.725964345 podStartE2EDuration="1.725964345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:41:26.723833349 +0000 UTC m=+2160.621508346" watchObservedRunningTime="2026-07-26 15:41:26.725964345 +0000 UTC m=+2160.623639342" Jul 26 15:41:29.027483 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:29.027444 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:41:35.749084 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:35.749028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:35.749646 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:35.749111 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:35.763130 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:35.763104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:35.774697 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:35.774673 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:41:47.935118 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:41:47.935084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:42:09.735233 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:09.735187 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:42:11.153003 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.152961 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:42:11.153528 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.153216 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" podUID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" containerName="payload-pre-processing" containerID="cri-o://9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442" gracePeriod=30 Jul 26 15:42:11.425997 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.425966 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:42:11.493586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.493545 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume\") pod \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " Jul 26 15:42:11.493761 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.493658 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkr92\" (UniqueName: \"kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92\") pod \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\" (UID: \"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f\") " Jul 26 15:42:11.493947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.493925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" (UID: "a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:42:11.494038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.494020 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:11.495959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.495932 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92" (OuterVolumeSpecName: "kube-api-access-qkr92") pod "a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" (UID: "a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f"). InnerVolumeSpecName "kube-api-access-qkr92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:11.595519 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.595482 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkr92\" (UniqueName: \"kubernetes.io/projected/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f-kube-api-access-qkr92\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:11.956196 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.956155 2575 generic.go:356] "Generic (PLEG): container finished" podID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" containerID="9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442" exitCode=0 Jul 26 15:42:11.956390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.956202 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" Jul 26 15:42:11.956390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.956242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" event={"ID":"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f","Type":"ContainerDied","Data":"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442"} Jul 26 15:42:11.956390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.956312 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws" event={"ID":"a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f","Type":"ContainerDied","Data":"9ef9a74b0990f93a90b956bba332ee6250c12be615a4838768743dbf79dee5ce"} Jul 26 15:42:11.956390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.956338 2575 scope.go:117] "RemoveContainer" containerID="9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442" Jul 26 15:42:11.965695 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.965676 2575 scope.go:117] "RemoveContainer" containerID="9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442" Jul 26 15:42:11.965951 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:42:11.965932 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442\": container with ID starting with 9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442 not found: ID does not exist" containerID="9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442" Jul 26 15:42:11.966012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.965960 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442"} err="failed to get container status \"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442\": rpc error: code = NotFound desc = could not find container \"9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442\": container with ID starting with 9029bc4cdafbae5b570824c53a6999c7901a4be32c4a58f7d5f13805b8ec2442 not found: ID does not exist" Jul 26 15:42:11.977912 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.977886 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:42:11.983614 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:11.983588 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-52d355e1-75f889f4b8-946ws"] Jul 26 15:42:12.554520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:12.554478 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" path="/var/lib/kubelet/pods/a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f/volumes" Jul 26 15:42:17.533773 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:17.533734 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:42:23.889304 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:23.889257 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:42:26.837464 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:26.837425 2575 scope.go:117] "RemoveContainer" containerID="53103554c4fd3861baab93ebc25966bf07cc1069233339d40d88b27597233021" Jul 26 15:42:38.909604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:38.909556 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" podUID="ae2052ff-c656-4804-891a-8c17ca56f0eb" containerName="main" containerID="cri-o://cdb0448ce240447fe726e22af9042d8a8fc0449b1dc754807ae509439f76e94a" gracePeriod=45 Jul 26 15:42:39.115595 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.115562 2575 generic.go:356] "Generic (PLEG): container finished" podID="ae2052ff-c656-4804-891a-8c17ca56f0eb" containerID="cdb0448ce240447fe726e22af9042d8a8fc0449b1dc754807ae509439f76e94a" exitCode=0 Jul 26 15:42:39.115762 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.115635 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" event={"ID":"ae2052ff-c656-4804-891a-8c17ca56f0eb","Type":"ContainerDied","Data":"cdb0448ce240447fe726e22af9042d8a8fc0449b1dc754807ae509439f76e94a"} Jul 26 15:42:39.177178 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.177148 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:42:39.359512 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf78m\" (UniqueName: \"kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359534 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359713 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359697 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359935 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache\") pod \"ae2052ff-c656-4804-891a-8c17ca56f0eb\" (UID: \"ae2052ff-c656-4804-891a-8c17ca56f0eb\") " Jul 26 15:42:39.359935 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359861 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home" (OuterVolumeSpecName: "home") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:39.359935 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:39.360064 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.359975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:39.360145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.360129 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-model-cache\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:39.360184 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.360150 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-tmp-dir\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:39.360184 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.360161 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-home\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:39.362075 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.362045 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:42:39.362344 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.362321 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm" (OuterVolumeSpecName: "dshm") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:39.362442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.362362 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m" (OuterVolumeSpecName: "kube-api-access-gf78m") pod "ae2052ff-c656-4804-891a-8c17ca56f0eb" (UID: "ae2052ff-c656-4804-891a-8c17ca56f0eb"). InnerVolumeSpecName "kube-api-access-gf78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:39.461122 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.461087 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2052ff-c656-4804-891a-8c17ca56f0eb-dshm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:39.461122 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.461120 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2052ff-c656-4804-891a-8c17ca56f0eb-tls-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:39.461369 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:39.461131 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf78m\" (UniqueName: \"kubernetes.io/projected/ae2052ff-c656-4804-891a-8c17ca56f0eb-kube-api-access-gf78m\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:42:40.122649 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.122561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" Jul 26 15:42:40.122649 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.122602 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp" event={"ID":"ae2052ff-c656-4804-891a-8c17ca56f0eb","Type":"ContainerDied","Data":"61a9befe4ee1e2afd5506453c839122acf8963b293753496c0eeec6ae4e76091"} Jul 26 15:42:40.123152 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.122655 2575 scope.go:117] "RemoveContainer" containerID="cdb0448ce240447fe726e22af9042d8a8fc0449b1dc754807ae509439f76e94a" Jul 26 15:42:40.144891 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.144863 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:42:40.148191 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.148164 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-52d355e1/e2e-lifecycle-model-52d355e1-kserve-65646b8b9f-wlvvp"] Jul 26 15:42:40.554671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:42:40.554631 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2052ff-c656-4804-891a-8c17ca56f0eb" path="/var/lib/kubelet/pods/ae2052ff-c656-4804-891a-8c17ca56f0eb/volumes" Jul 26 15:43:12.339300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:12.339193 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:43:12.339906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:12.339460 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" containerID="cri-o://0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54" gracePeriod=30 Jul 26 15:43:16.139103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:16.139060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:43:17.613632 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.613558 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:43:17.718104 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718067 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718130 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718177 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7rz7\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718377 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718412 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718450 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket\") pod \"def4ddff-e42f-4937-98dd-7f97d4ba5302\" (UID: \"def4ddff-e42f-4937-98dd-7f97d4ba5302\") " Jul 26 15:43:17.718770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718541 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:43:17.718770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718591 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:43:17.718770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718646 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:43:17.718940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718792 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.718940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718814 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.718940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718831 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def4ddff-e42f-4937-98dd-7f97d4ba5302-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.718940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.718791 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:43:17.719384 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.719362 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data" (OuterVolumeSpecName: "istio-data") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:43:17.720547 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.720528 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:43:17.720823 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.720804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:43:17.721078 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.721062 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token" (OuterVolumeSpecName: "istio-token") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:43:17.721131 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.721080 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7" (OuterVolumeSpecName: "kube-api-access-k7rz7") pod "def4ddff-e42f-4937-98dd-7f97d4ba5302" (UID: "def4ddff-e42f-4937-98dd-7f97d4ba5302"). InnerVolumeSpecName "kube-api-access-k7rz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:43:17.819350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819306 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rz7\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-kube-api-access-k7rz7\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.819350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819345 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.819350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819355 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.819602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819363 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def4ddff-e42f-4937-98dd-7f97d4ba5302-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.819602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819371 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:17.819602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:17.819378 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def4ddff-e42f-4937-98dd-7f97d4ba5302-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:43:18.332308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.329988 2575 generic.go:356] "Generic (PLEG): container finished" podID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerID="0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54" exitCode=0 Jul 26 15:43:18.332308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.330205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerDied","Data":"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54"} Jul 26 15:43:18.332308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.330329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" event={"ID":"def4ddff-e42f-4937-98dd-7f97d4ba5302","Type":"ContainerDied","Data":"48c7bd2dd27d5d50f523b7885de9708076bf2fcd90de9931cd3aa89c89a9337f"} Jul 26 15:43:18.332308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.330371 2575 scope.go:117] "RemoveContainer" containerID="0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54" Jul 26 15:43:18.332308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.330592 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq" Jul 26 15:43:18.342321 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.342292 2575 scope.go:117] "RemoveContainer" containerID="1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784" Jul 26 15:43:18.351485 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.351464 2575 scope.go:117] "RemoveContainer" containerID="0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54" Jul 26 15:43:18.351777 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:43:18.351755 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54\": container with ID starting with 0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54 not found: ID does not exist" containerID="0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54" Jul 26 15:43:18.351835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.351788 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54"} err="failed to get container status \"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54\": rpc error: code = NotFound desc = could not find container \"0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54\": container with ID starting with 0dfc3ad19420f0976aa727a58c63bf02a6bc7ebeecd53a5092ccc1a70f5d9b54 not found: ID does not exist" Jul 26 15:43:18.351835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.351808 2575 scope.go:117] "RemoveContainer" containerID="1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784" Jul 26 15:43:18.352050 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:43:18.352029 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784\": container with ID starting with 1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784 not found: ID does not exist" containerID="1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784" Jul 26 15:43:18.352093 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.352057 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784"} err="failed to get container status \"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784\": rpc error: code = NotFound desc = could not find container \"1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784\": container with ID starting with 1126365d42efb507b2b7e83bc61c8ef1f769266bbcca767e19269487e1f04784 not found: ID does not exist" Jul 26 15:43:18.358608 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.358582 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:43:18.362293 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.362252 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-52d355e1-openshift-default-6768877cfb-jg5jq"] Jul 26 15:43:18.555315 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:18.555264 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" path="/var/lib/kubelet/pods/def4ddff-e42f-4937-98dd-7f97d4ba5302/volumes" Jul 26 15:43:26.864829 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:26.864798 2575 scope.go:117] "RemoveContainer" containerID="df1b0bd0aa2ceaf7a6ab45295ff10048feb594697b58fc693ed7f277da78b0a4" Jul 26 15:43:28.944867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:43:28.944828 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:44:07.063686 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.063647 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064208 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064230 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064258 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" containerName="payload-pre-processing" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064265 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" containerName="payload-pre-processing" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064294 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064303 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064313 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2052ff-c656-4804-891a-8c17ca56f0eb" containerName="main" Jul 26 15:44:07.064328 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064321 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2052ff-c656-4804-891a-8c17ca56f0eb" containerName="main" Jul 26 15:44:07.064755 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064418 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064755 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064430 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2052ff-c656-4804-891a-8c17ca56f0eb" containerName="main" Jul 26 15:44:07.064755 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064441 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="def4ddff-e42f-4937-98dd-7f97d4ba5302" containerName="istio-proxy" Jul 26 15:44:07.064755 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.064450 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10bf91b-1c01-4aa7-ae02-20c8a9d8b68f" containerName="payload-pre-processing" Jul 26 15:44:07.067818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.067800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.070587 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.070564 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e47ce-openshift-default-dockercfg-cbrrb\"" Jul 26 15:44:07.078733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.078703 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:44:07.161737 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.161940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.161940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.161940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.161940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkk4\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.162181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.161960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.162181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.162006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.162181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.162032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.162181 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.162092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkk4\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263718 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.263859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.263849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.264190 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.264172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.264309 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.264266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.265648 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.265624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.266051 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.266035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.272106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.272080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.272325 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.272302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkk4\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4\") pod \"e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.380455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.380365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:07.547702 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:44:07.547670 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a84f6c_7a8d_46cb_85c4_c084107d29e0.slice/crio-fce579f996c1f20f325bd43fff8879b3bdd605023aac3267ce66a631c47ed1c7 WatchSource:0}: Error finding container fce579f996c1f20f325bd43fff8879b3bdd605023aac3267ce66a631c47ed1c7: Status 404 returned error can't find the container with id fce579f996c1f20f325bd43fff8879b3bdd605023aac3267ce66a631c47ed1c7 Jul 26 15:44:07.548858 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.548835 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:44:07.550046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.550003 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:07.550125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.550107 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:07.550174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.550157 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:07.604036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:07.603992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerStarted","Data":"fce579f996c1f20f325bd43fff8879b3bdd605023aac3267ce66a631c47ed1c7"} Jul 26 15:44:08.610662 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:08.610626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerStarted","Data":"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a"} Jul 26 15:44:08.632388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:08.632331 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podStartSLOduration=1.632314869 podStartE2EDuration="1.632314869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:08.630794105 +0000 UTC m=+2322.528469120" watchObservedRunningTime="2026-07-26 15:44:08.632314869 +0000 UTC m=+2322.529989866" Jul 26 15:44:09.380733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:09.380691 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:09.386132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:09.386100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:09.615712 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:09.615674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:09.616606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:09.616581 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:11.454247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.454208 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:44:11.457798 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.457780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.460540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.460511 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e47ce\"" Jul 26 15:44:11.460671 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.460610 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e47ce-dockercfg-fcbcs\"" Jul 26 15:44:11.470453 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.470418 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:44:11.504910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.504876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.504910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.504913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664gs\" (UniqueName: \"kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.605457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.605416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.605457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.605454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664gs\" (UniqueName: \"kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.606121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.606100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.613983 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.613957 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664gs\" (UniqueName: \"kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs\") pod \"payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.769977 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.769886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:11.916718 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:11.916685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:44:11.918562 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:44:11.918526 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ddb48f_522c_42e0_81b9_cb86488d53f6.slice/crio-3e87b5f9249cbe9ff43c52bef7ad3e734f44d01784ab00bd0ee75ffcddb2b7b6 WatchSource:0}: Error finding container 3e87b5f9249cbe9ff43c52bef7ad3e734f44d01784ab00bd0ee75ffcddb2b7b6: Status 404 returned error can't find the container with id 3e87b5f9249cbe9ff43c52bef7ad3e734f44d01784ab00bd0ee75ffcddb2b7b6 Jul 26 15:44:12.633656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:12.633624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" event={"ID":"38ddb48f-522c-42e0-81b9-cb86488d53f6","Type":"ContainerStarted","Data":"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1"} Jul 26 15:44:12.634107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:12.633676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" event={"ID":"38ddb48f-522c-42e0-81b9-cb86488d53f6","Type":"ContainerStarted","Data":"3e87b5f9249cbe9ff43c52bef7ad3e734f44d01784ab00bd0ee75ffcddb2b7b6"} Jul 26 15:44:12.634107 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:12.633706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:12.649579 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:12.649527 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" podStartSLOduration=1.649511901 podStartE2EDuration="1.649511901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:12.649095859 +0000 UTC m=+2326.546770857" watchObservedRunningTime="2026-07-26 15:44:12.649511901 +0000 UTC m=+2326.547186899" Jul 26 15:44:15.628907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.628874 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:44:15.632804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.632780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.635427 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.635400 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-28fa20-openshift-default-dockercfg-4nrgl\"" Jul 26 15:44:15.644388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.644363 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:44:15.742502 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742687 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742687 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7p5f\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742687 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742849 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742849 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742849 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.742963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.742934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843486 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843486 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843708 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843708 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843708 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7p5f\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.843985 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.844223 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.843999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.844223 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.844156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.844359 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.844243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.844400 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.844372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.846162 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.846133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.846299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.846212 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.851464 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.851443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.851833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.851812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7p5f\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f\") pod \"e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:15.944785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:15.944747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:16.102222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.102194 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:44:16.104184 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:44:16.104149 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359a21d9_0163_42fe_8ba5_65bdd9cbfaeb.slice/crio-7fa875adcabaa237bd97c1ac2430cec3ccd5e48fbb1a891445c77d9acebe8f4f WatchSource:0}: Error finding container 7fa875adcabaa237bd97c1ac2430cec3ccd5e48fbb1a891445c77d9acebe8f4f: Status 404 returned error can't find the container with id 7fa875adcabaa237bd97c1ac2430cec3ccd5e48fbb1a891445c77d9acebe8f4f Jul 26 15:44:16.106389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.106355 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:16.106470 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.106423 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:16.106470 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.106452 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:16.628733 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.628699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:44:16.658018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.657986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerStarted","Data":"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307"} Jul 26 15:44:16.658415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.658041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerStarted","Data":"7fa875adcabaa237bd97c1ac2430cec3ccd5e48fbb1a891445c77d9acebe8f4f"} Jul 26 15:44:16.680804 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.680734 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podStartSLOduration=1.6807123019999999 podStartE2EDuration="1.680712302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:16.678576832 +0000 UTC m=+2330.576251829" watchObservedRunningTime="2026-07-26 15:44:16.680712302 +0000 UTC m=+2330.578387301" Jul 26 15:44:16.945523 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:16.945488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:17.950836 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:17.950801 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:18.669767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:18.669736 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:18.670809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:18.670782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:19.392301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.391258 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:44:19.397311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.395683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.402297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.398839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-28fa20-dockercfg-djm44\"" Jul 26 15:44:19.402297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.399196 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-28fa20\"" Jul 26 15:44:19.408753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.408726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:44:19.425730 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.425700 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:44:19.429138 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.429119 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.436723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.436459 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:44:19.492300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.491747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.492300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.491825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm77d\" (UniqueName: \"kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.492300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.491861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn6v\" (UniqueName: \"kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.492300 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.491945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.593480 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.593431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.593750 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.593724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm77d\" (UniqueName: \"kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.593881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.593865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn6v\" (UniqueName: \"kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.594076 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.594060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.594638 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.594611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.595098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.595072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.604727 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.604671 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn6v\" (UniqueName: \"kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v\") pod \"payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.607267 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.607234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm77d\" (UniqueName: \"kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d\") pod \"payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.715774 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.715742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:19.741093 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.741051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:19.910240 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.910037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:44:19.960907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:19.960876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:44:19.961413 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:44:19.961389 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ddf18_31aa_4b9e_b464_08bb4e717821.slice/crio-befaad8db6fa29346ce6e0f2dc11c9ed47ff9026a2f791144419868eee9915ad WatchSource:0}: Error finding container befaad8db6fa29346ce6e0f2dc11c9ed47ff9026a2f791144419868eee9915ad: Status 404 returned error can't find the container with id befaad8db6fa29346ce6e0f2dc11c9ed47ff9026a2f791144419868eee9915ad Jul 26 15:44:20.690187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.690144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" event={"ID":"fe5ddf18-31aa-4b9e-b464-08bb4e717821","Type":"ContainerStarted","Data":"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203"} Jul 26 15:44:20.691353 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.691330 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:20.691561 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.691541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" event={"ID":"fe5ddf18-31aa-4b9e-b464-08bb4e717821","Type":"ContainerStarted","Data":"befaad8db6fa29346ce6e0f2dc11c9ed47ff9026a2f791144419868eee9915ad"} Jul 26 15:44:20.696296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.694404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" event={"ID":"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee","Type":"ContainerStarted","Data":"5d2c2ad58c3dfbc7bf56e6f1dfaee227a0e9556b1c094e6bb6c0334fb43ae8c6"} Jul 26 15:44:20.696296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.694454 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:20.696296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.694479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" event={"ID":"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee","Type":"ContainerStarted","Data":"475a200690e3b058b181133861d820057d5fc777d53344be8af91f7e102e8a01"} Jul 26 15:44:20.724228 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.724162 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" podStartSLOduration=1.7241389599999999 podStartE2EDuration="1.72413896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:20.720669301 +0000 UTC m=+2334.618344341" watchObservedRunningTime="2026-07-26 15:44:20.72413896 +0000 UTC m=+2334.621813960" Jul 26 15:44:20.737654 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:20.737597 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" podStartSLOduration=1.737581263 podStartE2EDuration="1.737581263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:44:20.736011199 +0000 UTC m=+2334.633686222" watchObservedRunningTime="2026-07-26 15:44:20.737581263 +0000 UTC m=+2334.635256261" Jul 26 15:44:23.642098 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:23.642065 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:44:24.631055 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.629584 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:52760->10.134.0.64:15021: read: connection reset by peer" start-of-body= Jul 26 15:44:24.631055 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.629644 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": read tcp 10.134.0.2:52760->10.134.0.64:15021: read: connection reset by peer" Jul 26 15:44:24.717630 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.717594 2575 generic.go:356] "Generic (PLEG): container finished" podID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerID="42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a" exitCode=0 Jul 26 15:44:24.718009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.717668 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerDied","Data":"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a"} Jul 26 15:44:24.718102 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.718088 2575 scope.go:117] "RemoveContainer" containerID="42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a" Jul 26 15:44:24.719229 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.719201 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:24.719327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.719312 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:24.719380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:24.719351 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:25.733604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:25.733555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerStarted","Data":"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b"} Jul 26 15:44:26.380772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:26.380722 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:26.380989 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:26.380859 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 15:44:26.380989 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:26.380911 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 15:44:27.381176 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:27.381125 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 15:44:27.381699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:27.381196 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 15:44:28.380915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:28.380862 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 15:44:28.381136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:28.380938 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 15:44:29.381775 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:29.381732 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 26 15:44:29.382292 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:29.381806 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 26 15:44:30.385682 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:30.385648 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:30.386172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:30.386053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:30.386744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:30.386723 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:44:31.700627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:31.700594 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:44:31.701103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:31.701032 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:44:32.744998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.744961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:44:32.789542 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.789507 2575 generic.go:356] "Generic (PLEG): container finished" podID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerID="71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307" exitCode=0 Jul 26 15:44:32.789700 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.789545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerDied","Data":"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307"} Jul 26 15:44:32.789923 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.789908 2575 scope.go:117] "RemoveContainer" containerID="71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307" Jul 26 15:44:32.790933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.790901 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:32.791039 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.790987 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:32.791104 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:32.791049 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:44:33.670056 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:33.670018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:33.801791 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:33.801749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerStarted","Data":"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878"} Jul 26 15:44:33.945775 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:33.945737 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:33.947050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:33.947023 2575 patch_prober.go:28] interesting pod/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 15:44:33.947177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:33.947084 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 15:44:34.946114 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:34.946072 2575 patch_prober.go:28] interesting pod/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 15:44:34.946558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:34.946130 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 15:44:35.945361 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:35.945326 2575 patch_prober.go:28] interesting pod/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 15:44:35.945542 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:35.945381 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 15:44:36.945903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:36.945861 2575 patch_prober.go:28] interesting pod/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 26 15:44:36.946344 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:36.945921 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 26 15:44:37.949440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:37.949407 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:37.949846 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:37.949676 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:37.950338 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:37.950322 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:44:47.534697 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:44:47.534652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:45:02.731386 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.731344 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:45:02.734999 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.734974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.737683 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.737658 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28fa20\"/\"kube-root-ca.crt\"" Jul 26 15:45:02.738814 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.738797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28fa20\"/\"test-model-28fa20-kserve-self-signed-certs\"" Jul 26 15:45:02.738905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.738824 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-28fa20\"/\"default-dockercfg-qvjk2\"" Jul 26 15:45:02.738905 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.738856 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-28fa20\"/\"openshift-service-ca.crt\"" Jul 26 15:45:02.744794 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.744773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:45:02.834685 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.834866 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.834866 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.834866 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.834981 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.834981 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.834893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gch\" (UniqueName: \"kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935732 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.935924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.935883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gch\" (UniqueName: \"kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.936247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.936204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.936247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.936229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.936441 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.936334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.938344 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.938325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.938504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.938488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:02.944311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:02.944265 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gch\" (UniqueName: \"kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch\") pod \"test-model-28fa20-kserve-6b7d45656c-dmpjz\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:03.045505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:03.045409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:03.193358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:03.193328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:45:03.195188 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:45:03.195159 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05d07b05_b2f1_4444_97cd_c183d9e52c87.slice/crio-8958e3fd264533ce3d65c8426a7bb936724402153a518a45f6fe6c67d900b3da WatchSource:0}: Error finding container 8958e3fd264533ce3d65c8426a7bb936724402153a518a45f6fe6c67d900b3da: Status 404 returned error can't find the container with id 8958e3fd264533ce3d65c8426a7bb936724402153a518a45f6fe6c67d900b3da Jul 26 15:45:03.977051 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:03.977007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" event={"ID":"05d07b05-b2f1-4444-97cd-c183d9e52c87","Type":"ContainerStarted","Data":"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c"} Jul 26 15:45:03.977487 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:03.977065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" event={"ID":"05d07b05-b2f1-4444-97cd-c183d9e52c87","Type":"ContainerStarted","Data":"8958e3fd264533ce3d65c8426a7bb936724402153a518a45f6fe6c67d900b3da"} Jul 26 15:45:03.996263 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:03.996203 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" podStartSLOduration=1.996186692 podStartE2EDuration="1.996186692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:03.9936343 +0000 UTC m=+2377.891309297" watchObservedRunningTime="2026-07-26 15:45:03.996186692 +0000 UTC m=+2377.893861707" Jul 26 15:45:07.133739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:07.133701 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:45:13.046409 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:13.046368 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:13.046409 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:13.046418 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:13.059269 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:13.059239 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:14.045313 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:14.045249 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:45:21.734499 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:21.734459 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:45:30.576460 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:30.576424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:45:30.581070 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:30.581044 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:45:53.819937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:45:53.819845 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:46:00.331519 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:00.331481 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:46:04.515451 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.515413 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:46:04.516368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.516261 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" podUID="38ddb48f-522c-42e0-81b9-cb86488d53f6" containerName="payload-pre-processing" containerID="cri-o://22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1" gracePeriod=30 Jul 26 15:46:04.787356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.787323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:46:04.833561 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.833519 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume\") pod \"38ddb48f-522c-42e0-81b9-cb86488d53f6\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " Jul 26 15:46:04.833561 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.833555 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-664gs\" (UniqueName: \"kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs\") pod \"38ddb48f-522c-42e0-81b9-cb86488d53f6\" (UID: \"38ddb48f-522c-42e0-81b9-cb86488d53f6\") " Jul 26 15:46:04.833917 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.833891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38ddb48f-522c-42e0-81b9-cb86488d53f6" (UID: "38ddb48f-522c-42e0-81b9-cb86488d53f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:04.835851 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.835824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs" (OuterVolumeSpecName: "kube-api-access-664gs") pod "38ddb48f-522c-42e0-81b9-cb86488d53f6" (UID: "38ddb48f-522c-42e0-81b9-cb86488d53f6"). InnerVolumeSpecName "kube-api-access-664gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:04.934258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.934218 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb48f-522c-42e0-81b9-cb86488d53f6-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:04.934258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:04.934252 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664gs\" (UniqueName: \"kubernetes.io/projected/38ddb48f-522c-42e0-81b9-cb86488d53f6-kube-api-access-664gs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:05.336826 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.336792 2575 generic.go:356] "Generic (PLEG): container finished" podID="38ddb48f-522c-42e0-81b9-cb86488d53f6" containerID="22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1" exitCode=0 Jul 26 15:46:05.337024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.336846 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" Jul 26 15:46:05.337024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.336854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" event={"ID":"38ddb48f-522c-42e0-81b9-cb86488d53f6","Type":"ContainerDied","Data":"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1"} Jul 26 15:46:05.337024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.336920 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq" event={"ID":"38ddb48f-522c-42e0-81b9-cb86488d53f6","Type":"ContainerDied","Data":"3e87b5f9249cbe9ff43c52bef7ad3e734f44d01784ab00bd0ee75ffcddb2b7b6"} Jul 26 15:46:05.337024 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.336941 2575 scope.go:117] "RemoveContainer" containerID="22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1" Jul 26 15:46:05.347197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.347168 2575 scope.go:117] "RemoveContainer" containerID="22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1" Jul 26 15:46:05.347480 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:05.347456 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1\": container with ID starting with 22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1 not found: ID does not exist" containerID="22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1" Jul 26 15:46:05.347554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.347488 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1"} err="failed to get container status \"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1\": rpc error: code = NotFound desc = could not find container \"22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1\": container with ID starting with 22fefd488d88db0dc69ec20f08e27c2a15a73aed330e0655e1fa618d0d2c48f1 not found: ID does not exist" Jul 26 15:46:05.360088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.360058 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:46:05.365651 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:05.365627 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e47ce-85f488c964-dxsnq"] Jul 26 15:46:06.555902 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:06.555863 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ddb48f-522c-42e0-81b9-cb86488d53f6" path="/var/lib/kubelet/pods/38ddb48f-522c-42e0-81b9-cb86488d53f6/volumes" Jul 26 15:46:08.842550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:08.842504 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" podUID="05d07b05-b2f1-4444-97cd-c183d9e52c87" containerName="main" containerID="cri-o://f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c" gracePeriod=45 Jul 26 15:46:09.105770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.105707 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:46:09.175021 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.174986 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175062 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175137 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175154 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175187 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54gch\" (UniqueName: \"kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch\") pod \"05d07b05-b2f1-4444-97cd-c183d9e52c87\" (UID: \"05d07b05-b2f1-4444-97cd-c183d9e52c87\") " Jul 26 15:46:09.175535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache" (OuterVolumeSpecName: "model-cache") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:09.175535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175413 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home" (OuterVolumeSpecName: "home") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:09.175633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:09.175633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175563 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-model-cache\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.175633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.175584 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-home\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.177768 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.177732 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:46:09.178018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.177990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch" (OuterVolumeSpecName: "kube-api-access-54gch") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "kube-api-access-54gch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:09.178208 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.178190 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm" (OuterVolumeSpecName: "dshm") pod "05d07b05-b2f1-4444-97cd-c183d9e52c87" (UID: "05d07b05-b2f1-4444-97cd-c183d9e52c87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:09.277071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.277015 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-dshm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.277071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.277063 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05d07b05-b2f1-4444-97cd-c183d9e52c87-tmp-dir\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.277071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.277075 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05d07b05-b2f1-4444-97cd-c183d9e52c87-tls-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.277071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.277085 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gch\" (UniqueName: \"kubernetes.io/projected/05d07b05-b2f1-4444-97cd-c183d9e52c87-kube-api-access-54gch\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:09.367873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.367784 2575 generic.go:356] "Generic (PLEG): container finished" podID="05d07b05-b2f1-4444-97cd-c183d9e52c87" containerID="f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c" exitCode=0 Jul 26 15:46:09.367873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.367859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" event={"ID":"05d07b05-b2f1-4444-97cd-c183d9e52c87","Type":"ContainerDied","Data":"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c"} Jul 26 15:46:09.368050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.367887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" event={"ID":"05d07b05-b2f1-4444-97cd-c183d9e52c87","Type":"ContainerDied","Data":"8958e3fd264533ce3d65c8426a7bb936724402153a518a45f6fe6c67d900b3da"} Jul 26 15:46:09.368050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.367916 2575 scope.go:117] "RemoveContainer" containerID="f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c" Jul 26 15:46:09.368050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.368019 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz" Jul 26 15:46:09.382018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.381998 2575 scope.go:117] "RemoveContainer" containerID="f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c" Jul 26 15:46:09.382335 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:09.382310 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c\": container with ID starting with f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c not found: ID does not exist" containerID="f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c" Jul 26 15:46:09.382415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.382347 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c"} err="failed to get container status \"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c\": rpc error: code = NotFound desc = could not find container \"f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c\": container with ID starting with f8e3ac6b8eff9ed346da4f8c8453a0c0fbabfdf898106c001f6c8dd2ae34d06c not found: ID does not exist" Jul 26 15:46:09.392327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.392265 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:46:09.394605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.394582 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-28fa20/test-model-28fa20-kserve-6b7d45656c-dmpjz"] Jul 26 15:46:09.732800 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:09.732764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:46:10.555207 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:10.555169 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05d07b05-b2f1-4444-97cd-c183d9e52c87" path="/var/lib/kubelet/pods/05d07b05-b2f1-4444-97cd-c183d9e52c87/volumes" Jul 26 15:46:15.343456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:15.343421 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:46:15.343941 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:15.343698 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" containerID="cri-o://991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b" gracePeriod=30 Jul 26 15:46:15.388244 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:15.388202 2575 patch_prober.go:28] interesting pod/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:46:15.388458 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:15.388311 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:46:20.605240 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.605212 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:46:20.688226 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688226 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688231 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqkk4\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688318 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688340 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688382 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688436 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy\") pod \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\" (UID: \"73a84f6c-7a8d-46cb-85c4-c084107d29e0\") " Jul 26 15:46:20.688879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:20.688879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:20.688879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688761 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:20.689007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688879 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.689007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688896 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.689007 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.688928 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.689212 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.689187 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:20.689310 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.689291 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:20.690953 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.690926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:46:20.691318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.691260 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4" (OuterVolumeSpecName: "kube-api-access-vqkk4") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "kube-api-access-vqkk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:20.691691 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.691670 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:20.691768 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.691700 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "73a84f6c-7a8d-46cb-85c4-c084107d29e0" (UID: "73a84f6c-7a8d-46cb-85c4-c084107d29e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:20.789518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789473 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.789518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789512 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.789518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789521 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.789518 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789529 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.789799 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789537 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqkk4\" (UniqueName: \"kubernetes.io/projected/73a84f6c-7a8d-46cb-85c4-c084107d29e0-kube-api-access-vqkk4\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:20.789799 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:20.789546 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a84f6c-7a8d-46cb-85c4-c084107d29e0-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:21.440335 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.440270 2575 generic.go:356] "Generic (PLEG): container finished" podID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerID="991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b" exitCode=0 Jul 26 15:46:21.440509 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.440359 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" Jul 26 15:46:21.440509 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.440365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerDied","Data":"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b"} Jul 26 15:46:21.440509 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.440440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj" event={"ID":"73a84f6c-7a8d-46cb-85c4-c084107d29e0","Type":"ContainerDied","Data":"fce579f996c1f20f325bd43fff8879b3bdd605023aac3267ce66a631c47ed1c7"} Jul 26 15:46:21.440509 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.440464 2575 scope.go:117] "RemoveContainer" containerID="991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b" Jul 26 15:46:21.450418 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.450398 2575 scope.go:117] "RemoveContainer" containerID="42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a" Jul 26 15:46:21.459154 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.458990 2575 scope.go:117] "RemoveContainer" containerID="991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b" Jul 26 15:46:21.459320 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:21.459270 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b\": container with ID starting with 991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b not found: ID does not exist" containerID="991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b" Jul 26 15:46:21.459399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.459330 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b"} err="failed to get container status \"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b\": rpc error: code = NotFound desc = could not find container \"991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b\": container with ID starting with 991c2392d81b182c9db58bb2a16bec4d94bcef47b3ce8a79e6ac19c6ce852b2b not found: ID does not exist" Jul 26 15:46:21.459399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.459349 2575 scope.go:117] "RemoveContainer" containerID="42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a" Jul 26 15:46:21.459607 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:21.459576 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a\": container with ID starting with 42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a not found: ID does not exist" containerID="42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a" Jul 26 15:46:21.459676 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.459614 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a"} err="failed to get container status \"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a\": rpc error: code = NotFound desc = could not find container \"42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a\": container with ID starting with 42265ff631e6eeee4fa74452227ee56af1f8a20ff988d605ff0cf0e961c6757a not found: ID does not exist" Jul 26 15:46:21.462938 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.462917 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:46:21.466813 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:21.466782 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e47ce-openshift-default-6dfc89b448-t9qzj"] Jul 26 15:46:22.555540 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:22.555505 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" path="/var/lib/kubelet/pods/73a84f6c-7a8d-46cb-85c4-c084107d29e0/volumes" Jul 26 15:46:26.186503 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.186469 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:46:26.187081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.186659 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" podUID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" containerName="payload-processing" containerID="cri-o://2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203" gracePeriod=30 Jul 26 15:46:26.194593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.194568 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:46:26.194796 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.194774 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" podUID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" containerName="payload-pre-processing" containerID="cri-o://5d2c2ad58c3dfbc7bf56e6f1dfaee227a0e9556b1c094e6bb6c0334fb43ae8c6" gracePeriod=30 Jul 26 15:46:26.453413 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.453382 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:46:26.476828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.475500 2575 generic.go:356] "Generic (PLEG): container finished" podID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" containerID="2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203" exitCode=0 Jul 26 15:46:26.476828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.475813 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" Jul 26 15:46:26.476828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.476174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" event={"ID":"fe5ddf18-31aa-4b9e-b464-08bb4e717821","Type":"ContainerDied","Data":"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203"} Jul 26 15:46:26.476828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.476257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd" event={"ID":"fe5ddf18-31aa-4b9e-b464-08bb4e717821","Type":"ContainerDied","Data":"befaad8db6fa29346ce6e0f2dc11c9ed47ff9026a2f791144419868eee9915ad"} Jul 26 15:46:26.476828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.476313 2575 scope.go:117] "RemoveContainer" containerID="2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203" Jul 26 15:46:26.479575 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.479263 2575 generic.go:356] "Generic (PLEG): container finished" podID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" containerID="5d2c2ad58c3dfbc7bf56e6f1dfaee227a0e9556b1c094e6bb6c0334fb43ae8c6" exitCode=0 Jul 26 15:46:26.479697 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.479563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" event={"ID":"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee","Type":"ContainerDied","Data":"5d2c2ad58c3dfbc7bf56e6f1dfaee227a0e9556b1c094e6bb6c0334fb43ae8c6"} Jul 26 15:46:26.490455 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.490436 2575 scope.go:117] "RemoveContainer" containerID="2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203" Jul 26 15:46:26.490708 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:26.490687 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203\": container with ID starting with 2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203 not found: ID does not exist" containerID="2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203" Jul 26 15:46:26.490781 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.490720 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203"} err="failed to get container status \"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203\": rpc error: code = NotFound desc = could not find container \"2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203\": container with ID starting with 2a6ade2aa2dcf8f2359153bd87b13a0db53b696e657f4b1aae84fa7b7d0bf203 not found: ID does not exist" Jul 26 15:46:26.493915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.493890 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:46:26.542315 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.542268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpn6v\" (UniqueName: \"kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v\") pod \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " Jul 26 15:46:26.542511 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.542347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume\") pod \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\" (UID: \"fe5ddf18-31aa-4b9e-b464-08bb4e717821\") " Jul 26 15:46:26.542728 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.542704 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe5ddf18-31aa-4b9e-b464-08bb4e717821" (UID: "fe5ddf18-31aa-4b9e-b464-08bb4e717821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:26.544546 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.544522 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v" (OuterVolumeSpecName: "kube-api-access-dpn6v") pod "fe5ddf18-31aa-4b9e-b464-08bb4e717821" (UID: "fe5ddf18-31aa-4b9e-b464-08bb4e717821"). InnerVolumeSpecName "kube-api-access-dpn6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:26.642960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.642923 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm77d\" (UniqueName: \"kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d\") pod \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " Jul 26 15:46:26.643140 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.642987 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume\") pod \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\" (UID: \"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee\") " Jul 26 15:46:26.643185 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.643170 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ddf18-31aa-4b9e-b464-08bb4e717821-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:26.643220 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.643183 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn6v\" (UniqueName: \"kubernetes.io/projected/fe5ddf18-31aa-4b9e-b464-08bb4e717821-kube-api-access-dpn6v\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:26.643376 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.643344 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" (UID: "ed2c7eb8-c417-4d7f-b943-f70ade3f00ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:26.645182 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.645160 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d" (OuterVolumeSpecName: "kube-api-access-vm77d") pod "ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" (UID: "ed2c7eb8-c417-4d7f-b943-f70ade3f00ee"). InnerVolumeSpecName "kube-api-access-vm77d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:26.743763 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.743674 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm77d\" (UniqueName: \"kubernetes.io/projected/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-kube-api-access-vm77d\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:26.743763 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.743706 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:26.793106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.793072 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:46:26.796496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:26.796469 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-28fa20-67c955ff86-2nlhd"] Jul 26 15:46:27.486316 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:27.486194 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" Jul 26 15:46:27.486316 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:27.486201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw" event={"ID":"ed2c7eb8-c417-4d7f-b943-f70ade3f00ee","Type":"ContainerDied","Data":"475a200690e3b058b181133861d820057d5fc777d53344be8af91f7e102e8a01"} Jul 26 15:46:27.486865 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:27.486339 2575 scope.go:117] "RemoveContainer" containerID="5d2c2ad58c3dfbc7bf56e6f1dfaee227a0e9556b1c094e6bb6c0334fb43ae8c6" Jul 26 15:46:27.507855 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:27.507827 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:46:27.513739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:27.513715 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-28fa20-746b94cb4-ftkhw"] Jul 26 15:46:28.553754 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:28.553721 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" path="/var/lib/kubelet/pods/ed2c7eb8-c417-4d7f-b943-f70ade3f00ee/volumes" Jul 26 15:46:28.554108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:28.554079 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" path="/var/lib/kubelet/pods/fe5ddf18-31aa-4b9e-b464-08bb4e717821/volumes" Jul 26 15:46:29.231562 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:29.231525 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:46:37.114214 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.114177 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:46:37.114650 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.114435 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" containerID="cri-o://53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878" gracePeriod=30 Jul 26 15:46:37.952383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.952340 2575 patch_prober.go:28] interesting pod/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:46:37.952561 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.952485 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:46:37.971009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.970971 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:46:37.971543 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971523 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ddb48f-522c-42e0-81b9-cb86488d53f6" containerName="payload-pre-processing" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971548 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ddb48f-522c-42e0-81b9-cb86488d53f6" containerName="payload-pre-processing" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971561 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971569 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971588 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05d07b05-b2f1-4444-97cd-c183d9e52c87" containerName="main" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971595 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="05d07b05-b2f1-4444-97cd-c183d9e52c87" containerName="main" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971603 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" containerName="payload-pre-processing" Jul 26 15:46:37.971621 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971608 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" containerName="payload-pre-processing" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971623 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" containerName="payload-processing" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971634 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" containerName="payload-processing" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971690 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971700 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="05d07b05-b2f1-4444-97cd-c183d9e52c87" containerName="main" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ddb48f-522c-42e0-81b9-cb86488d53f6" containerName="payload-pre-processing" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971721 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2c7eb8-c417-4d7f-b943-f70ade3f00ee" containerName="payload-pre-processing" Jul 26 15:46:37.971859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.971728 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ddf18-31aa-4b9e-b464-08bb4e717821" containerName="payload-processing" Jul 26 15:46:37.974801 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.974780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:37.977431 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.977409 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2ad2c2-openshift-default-dockercfg-cg2s2\"" Jul 26 15:46:37.986960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:37.986932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:46:38.154358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tzq\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.154818 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.154778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256130 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256130 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256130 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tzq\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256130 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256740 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256740 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256884 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.256971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.256947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.257072 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.257007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.258599 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.258576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.258882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.258864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.268451 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.268425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.268557 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.268476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tzq\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq\") pod \"e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.288345 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.288311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:38.440353 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.440321 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:46:38.441659 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:46:38.441632 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d204d_9e90_4239_9de3_4e110b1f4554.slice/crio-ac8c638e49e4a3e8dc6646e263440f49d010644d7d7205435cd9f621c97cf8ea WatchSource:0}: Error finding container ac8c638e49e4a3e8dc6646e263440f49d010644d7d7205435cd9f621c97cf8ea: Status 404 returned error can't find the container with id ac8c638e49e4a3e8dc6646e263440f49d010644d7d7205435cd9f621c97cf8ea Jul 26 15:46:38.443888 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.443849 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:38.443957 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.443923 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:38.443957 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.443952 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:38.556121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.556036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerStarted","Data":"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1"} Jul 26 15:46:38.556121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.556092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerStarted","Data":"ac8c638e49e4a3e8dc6646e263440f49d010644d7d7205435cd9f621c97cf8ea"} Jul 26 15:46:38.574128 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:38.574059 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podStartSLOduration=1.57403731 podStartE2EDuration="1.57403731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:38.570223253 +0000 UTC m=+2472.467898253" watchObservedRunningTime="2026-07-26 15:46:38.57403731 +0000 UTC m=+2472.471712309" Jul 26 15:46:39.288792 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:39.288746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:39.294040 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:39.294010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:39.556393 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:39.556305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:39.557251 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:39.557235 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:41.407080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.407040 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:46:41.407505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.407490 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:41.407550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.407507 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:41.407582 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.407571 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a84f6c-7a8d-46cb-85c4-c084107d29e0" containerName="istio-proxy" Jul 26 15:46:41.410800 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.410777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.413924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.413903 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2ad2c2-dockercfg-hlq89\"" Jul 26 15:46:41.414120 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.414104 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2ad2c2\"" Jul 26 15:46:41.419341 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.419308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:46:41.481778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.481736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt925\" (UniqueName: \"kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.481957 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.481858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.582539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.582484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt925\" (UniqueName: \"kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.582744 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.582588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.583384 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.583355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.590882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.590852 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt925\" (UniqueName: \"kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925\") pod \"payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.722723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.722684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:41.869347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:41.869315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:46:41.870881 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:46:41.870848 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48fe5d4_f7dd_4560_aadd_07a6debf4f2f.slice/crio-92d9b722d8c34b81725be2707d59227e41d201825d49e8d6c9ba1b9b19b94aac WatchSource:0}: Error finding container 92d9b722d8c34b81725be2707d59227e41d201825d49e8d6c9ba1b9b19b94aac: Status 404 returned error can't find the container with id 92d9b722d8c34b81725be2707d59227e41d201825d49e8d6c9ba1b9b19b94aac Jul 26 15:46:42.361915 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.361890 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:46:42.390921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.390825 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7p5f\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.390921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.390863 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.390921 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.390884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.390937 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.390974 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391009 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391103 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.391206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391137 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert\") pod \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\" (UID: \"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb\") " Jul 26 15:46:42.392103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391799 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:42.392103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391895 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:42.392103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.391984 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:42.392377 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.392184 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:42.392435 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.392399 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:42.393861 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.393833 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:42.394570 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.394544 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:46:42.394666 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.394577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f" (OuterVolumeSpecName: "kube-api-access-f7p5f") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "kube-api-access-f7p5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:46:42.394867 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.394836 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" (UID: "359a21d9-0163-42fe-8ba5-65bdd9cbfaeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:46:42.491835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491790 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.491835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491826 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.491835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491839 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491850 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491862 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491873 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491887 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7p5f\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-kube-api-access-f7p5f\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491898 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.492337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.491911 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:46:42.572717 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.572686 2575 generic.go:356] "Generic (PLEG): container finished" podID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerID="53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878" exitCode=0 Jul 26 15:46:42.572882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.572756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" Jul 26 15:46:42.572882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.572776 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerDied","Data":"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878"} Jul 26 15:46:42.572882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.572828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr" event={"ID":"359a21d9-0163-42fe-8ba5-65bdd9cbfaeb","Type":"ContainerDied","Data":"7fa875adcabaa237bd97c1ac2430cec3ccd5e48fbb1a891445c77d9acebe8f4f"} Jul 26 15:46:42.572882 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.572851 2575 scope.go:117] "RemoveContainer" containerID="53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878" Jul 26 15:46:42.574683 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.574654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" event={"ID":"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f","Type":"ContainerStarted","Data":"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d"} Jul 26 15:46:42.574821 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.574716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" event={"ID":"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f","Type":"ContainerStarted","Data":"92d9b722d8c34b81725be2707d59227e41d201825d49e8d6c9ba1b9b19b94aac"} Jul 26 15:46:42.574890 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.574817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:42.584757 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.584722 2575 scope.go:117] "RemoveContainer" containerID="71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307" Jul 26 15:46:42.596362 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.595601 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:46:42.597752 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.597733 2575 scope.go:117] "RemoveContainer" containerID="53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878" Jul 26 15:46:42.598100 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:42.598073 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878\": container with ID starting with 53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878 not found: ID does not exist" containerID="53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878" Jul 26 15:46:42.598188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.598110 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878"} err="failed to get container status \"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878\": rpc error: code = NotFound desc = could not find container \"53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878\": container with ID starting with 53c27c1234c14b400928a4ea677e4e34d7dea908ecea48eee648f93550dfb878 not found: ID does not exist" Jul 26 15:46:42.598188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.598133 2575 scope.go:117] "RemoveContainer" containerID="71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307" Jul 26 15:46:42.598834 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:46:42.598804 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307\": container with ID starting with 71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307 not found: ID does not exist" containerID="71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307" Jul 26 15:46:42.598929 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.598840 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307"} err="failed to get container status \"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307\": rpc error: code = NotFound desc = could not find container \"71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307\": container with ID starting with 71ccbfd6772388fe839a03efcaf9e359f068affd620256963bd093db97219307 not found: ID does not exist" Jul 26 15:46:42.601349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.601312 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-28fa20-openshift-default-6b9b485f74-xdxhr"] Jul 26 15:46:42.614032 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:42.613982 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" podStartSLOduration=1.613968652 podStartE2EDuration="1.613968652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:42.613100395 +0000 UTC m=+2476.510775394" watchObservedRunningTime="2026-07-26 15:46:42.613968652 +0000 UTC m=+2476.511643651" Jul 26 15:46:44.555917 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:44.555886 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" path="/var/lib/kubelet/pods/359a21d9-0163-42fe-8ba5-65bdd9cbfaeb/volumes" Jul 26 15:46:45.684614 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.684580 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:46:45.685088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.685069 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:45.685151 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.685090 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:45.685207 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.685152 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:45.690184 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.690159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.692981 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.692956 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14ba5c-openshift-default-dockercfg-qgkwj\"" Jul 26 15:46:45.699721 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.699696 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:46:45.721340 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721340 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721627 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbgs\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.721907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.721666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.822747 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.822706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.822747 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.822750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.822979 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.822882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.822979 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.822928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823234 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823234 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823234 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbgs\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823418 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823418 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823672 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.823978 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.823672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.825506 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.825486 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.825770 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.825751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.834429 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.834403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbgs\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:45.834527 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:45.834408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token\") pod \"e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:46.003082 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.002989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:46.180058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.180027 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:46:46.181541 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:46:46.181506 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abc8891_3748_46f3_8cc8_37c5afa518d2.slice/crio-a0edee5f313913a7a03e8b6dc71a76e711b1044958bd90818c556e037d471c7e WatchSource:0}: Error finding container a0edee5f313913a7a03e8b6dc71a76e711b1044958bd90818c556e037d471c7e: Status 404 returned error can't find the container with id a0edee5f313913a7a03e8b6dc71a76e711b1044958bd90818c556e037d471c7e Jul 26 15:46:46.183714 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.183682 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:46.183782 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.183749 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:46.183782 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.183778 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:46.601922 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.601825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerStarted","Data":"4a72c0edaadb035a1528ac62fe9244fffb31d0a503b67642ced70f2d867a6da6"} Jul 26 15:46:46.601922 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.601887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerStarted","Data":"a0edee5f313913a7a03e8b6dc71a76e711b1044958bd90818c556e037d471c7e"} Jul 26 15:46:46.636657 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:46.636605 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podStartSLOduration=1.6365881089999998 podStartE2EDuration="1.636588109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:46.633732191 +0000 UTC m=+2480.531407199" watchObservedRunningTime="2026-07-26 15:46:46.636588109 +0000 UTC m=+2480.534263106" Jul 26 15:46:47.003240 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:47.003198 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:47.008398 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:47.008371 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:47.607453 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:47.607421 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:47.608388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:47.608371 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:46:48.555080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:48.555052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:46:49.413535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.413497 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:46:49.413934 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.413919 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:49.413976 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.413936 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:49.414018 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.414009 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="359a21d9-0163-42fe-8ba5-65bdd9cbfaeb" containerName="istio-proxy" Jul 26 15:46:49.417329 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.417309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.419924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.419903 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14ba5c-dockercfg-jtz8c\"" Jul 26 15:46:49.419924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.419915 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14ba5c\"" Jul 26 15:46:49.428112 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.428083 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:46:49.460999 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.460955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.461200 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.461042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wql5j\" (UniqueName: \"kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.561816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.561759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.562435 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.561884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wql5j\" (UniqueName: \"kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.562560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.562541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.570245 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.570214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wql5j\" (UniqueName: \"kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j\") pod \"payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.729119 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.729080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:49.875127 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:49.875099 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:46:49.876818 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:46:49.876774 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d8ba00_0c6d_4a67_9ff9_5647bc2618f3.slice/crio-579046defd209d0f67cb802b53a14db71bbd88e90a62ec9d97bb75978431af39 WatchSource:0}: Error finding container 579046defd209d0f67cb802b53a14db71bbd88e90a62ec9d97bb75978431af39: Status 404 returned error can't find the container with id 579046defd209d0f67cb802b53a14db71bbd88e90a62ec9d97bb75978431af39 Jul 26 15:46:50.625609 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:50.625564 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" event={"ID":"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3","Type":"ContainerStarted","Data":"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476"} Jul 26 15:46:50.626067 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:50.625628 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:46:50.626067 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:50.625647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" event={"ID":"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3","Type":"ContainerStarted","Data":"579046defd209d0f67cb802b53a14db71bbd88e90a62ec9d97bb75978431af39"} Jul 26 15:46:50.642236 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:50.642180 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" podStartSLOduration=1.642166174 podStartE2EDuration="1.642166174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:50.64089568 +0000 UTC m=+2484.538570678" watchObservedRunningTime="2026-07-26 15:46:50.642166174 +0000 UTC m=+2484.539841174" Jul 26 15:46:53.586445 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:53.585469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:46:55.597510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:55.597463 2575 patch_prober.go:28] interesting pod/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:46:55.597916 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:55.597541 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:46:57.667095 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.667007 2575 generic.go:356] "Generic (PLEG): container finished" podID="849d204d-9e90-4239-9de3-4e110b1f4554" containerID="f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1" exitCode=0 Jul 26 15:46:57.667095 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.667061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerDied","Data":"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1"} Jul 26 15:46:57.667555 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.667510 2575 scope.go:117] "RemoveContainer" containerID="f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1" Jul 26 15:46:57.668541 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.668511 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:57.668606 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.668579 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:57.668643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:57.668610 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:46:58.675093 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:58.675049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerStarted","Data":"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d"} Jul 26 15:46:59.289558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:59.289496 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:46:59.290090 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:59.290054 2575 patch_prober.go:28] interesting pod/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 26 15:46:59.290234 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:46:59.290135 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 26 15:47:00.288933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:00.288884 2575 patch_prober.go:28] interesting pod/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 26 15:47:00.289481 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:00.288966 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 26 15:47:01.289706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:01.289223 2575 patch_prober.go:28] interesting pod/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 26 15:47:01.289706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:01.289311 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 26 15:47:01.633661 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:01.633565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:47:02.289387 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:02.289347 2575 patch_prober.go:28] interesting pod/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Jul 26 15:47:02.289704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:02.289676 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Jul 26 15:47:03.292721 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:03.292685 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:47:03.293187 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:03.293070 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:47:03.293820 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:03.293801 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:47:05.132931 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:05.132888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:47:06.738717 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.738679 2575 generic.go:356] "Generic (PLEG): container finished" podID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerID="4a72c0edaadb035a1528ac62fe9244fffb31d0a503b67642ced70f2d867a6da6" exitCode=0 Jul 26 15:47:06.739151 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.738744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerDied","Data":"4a72c0edaadb035a1528ac62fe9244fffb31d0a503b67642ced70f2d867a6da6"} Jul 26 15:47:06.739151 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.739129 2575 scope.go:117] "RemoveContainer" containerID="4a72c0edaadb035a1528ac62fe9244fffb31d0a503b67642ced70f2d867a6da6" Jul 26 15:47:06.740175 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.740145 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:06.740230 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.740219 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:06.740265 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:06.740252 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:07.746053 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:07.746020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerStarted","Data":"6c0d20a866fe0035565b5c7f324b11b1fdaf269251b47a9093c87b0332cffbb4"} Jul 26 15:47:08.003597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:08.003507 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:47:08.003864 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:08.003574 2575 patch_prober.go:28] interesting pod/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 26 15:47:08.003950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:08.003922 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 26 15:47:09.004446 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:09.004410 2575 patch_prober.go:28] interesting pod/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 26 15:47:09.004957 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:09.004492 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 26 15:47:10.003799 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:10.003745 2575 patch_prober.go:28] interesting pod/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 26 15:47:10.003996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:10.003834 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 26 15:47:11.004016 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:11.003978 2575 patch_prober.go:28] interesting pod/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Jul 26 15:47:11.004577 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:11.004046 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Jul 26 15:47:12.006872 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:12.006840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:47:12.007308 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:12.007108 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:47:12.007924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:12.007902 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:47:16.707358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:16.707324 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:47:16.707856 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:16.707527 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" podUID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" containerName="payload-processing" containerID="cri-o://90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d" gracePeriod=30 Jul 26 15:47:16.967933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:16.967869 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:47:17.134956 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.134922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume\") pod \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " Jul 26 15:47:17.135143 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.135010 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt925\" (UniqueName: \"kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925\") pod \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\" (UID: \"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f\") " Jul 26 15:47:17.135410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.135383 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" (UID: "f48fe5d4-f7dd-4560-aadd-07a6debf4f2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:17.137216 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.137192 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925" (OuterVolumeSpecName: "kube-api-access-tt925") pod "f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" (UID: "f48fe5d4-f7dd-4560-aadd-07a6debf4f2f"). InnerVolumeSpecName "kube-api-access-tt925". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:17.235843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.235757 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:17.235843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.235787 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt925\" (UniqueName: \"kubernetes.io/projected/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f-kube-api-access-tt925\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:17.801785 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.801751 2575 generic.go:356] "Generic (PLEG): container finished" podID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" containerID="90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d" exitCode=0 Jul 26 15:47:17.802255 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.801803 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" event={"ID":"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f","Type":"ContainerDied","Data":"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d"} Jul 26 15:47:17.802255 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.801811 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" Jul 26 15:47:17.802255 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.801856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj" event={"ID":"f48fe5d4-f7dd-4560-aadd-07a6debf4f2f","Type":"ContainerDied","Data":"92d9b722d8c34b81725be2707d59227e41d201825d49e8d6c9ba1b9b19b94aac"} Jul 26 15:47:17.802255 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.801885 2575 scope.go:117] "RemoveContainer" containerID="90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d" Jul 26 15:47:17.812380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.812339 2575 scope.go:117] "RemoveContainer" containerID="90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d" Jul 26 15:47:17.812672 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:47:17.812652 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d\": container with ID starting with 90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d not found: ID does not exist" containerID="90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d" Jul 26 15:47:17.812739 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.812680 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d"} err="failed to get container status \"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d\": rpc error: code = NotFound desc = could not find container \"90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d\": container with ID starting with 90daac6ee8c81fe71c4e989db2f439c82ca1061dc5c07caba04dde95b239444d not found: ID does not exist" Jul 26 15:47:17.824542 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.824504 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:47:17.826794 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:17.826770 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2ad2c2-7976fcdd44-dzxpj"] Jul 26 15:47:18.561914 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:18.561867 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" path="/var/lib/kubelet/pods/f48fe5d4-f7dd-4560-aadd-07a6debf4f2f/volumes" Jul 26 15:47:21.632521 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:21.632476 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:47:27.281484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:27.281383 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:47:27.281964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:27.281651 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" containerID="cri-o://bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d" gracePeriod=30 Jul 26 15:47:32.541937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.541872 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:47:32.575247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575284 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575307 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575393 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54tzq\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575642 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575695 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:32.575723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:32.575831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575800 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.575831 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575799 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:32.575986 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.575837 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo\") pod \"849d204d-9e90-4239-9de3-4e110b1f4554\" (UID: \"849d204d-9e90-4239-9de3-4e110b1f4554\") " Jul 26 15:47:32.576050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576034 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:32.576118 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576088 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data" (OuterVolumeSpecName: "istio-data") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:32.576305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576255 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d204d-9e90-4239-9de3-4e110b1f4554-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.576305 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576302 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.576609 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576317 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.576609 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576333 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.576609 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.576348 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.577988 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.577962 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:32.578184 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.578165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq" (OuterVolumeSpecName: "kube-api-access-54tzq") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "kube-api-access-54tzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:32.578613 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.578596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token" (OuterVolumeSpecName: "istio-token") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:32.578692 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.578594 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "849d204d-9e90-4239-9de3-4e110b1f4554" (UID: "849d204d-9e90-4239-9de3-4e110b1f4554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:47:32.676958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.676917 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.676958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.676950 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tzq\" (UniqueName: \"kubernetes.io/projected/849d204d-9e90-4239-9de3-4e110b1f4554-kube-api-access-54tzq\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.676958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.676959 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d204d-9e90-4239-9de3-4e110b1f4554-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.677201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.676970 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d204d-9e90-4239-9de3-4e110b1f4554-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:32.886242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.886135 2575 generic.go:356] "Generic (PLEG): container finished" podID="849d204d-9e90-4239-9de3-4e110b1f4554" containerID="bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d" exitCode=0 Jul 26 15:47:32.886242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.886206 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" Jul 26 15:47:32.886242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.886214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerDied","Data":"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d"} Jul 26 15:47:32.886539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.886306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq" event={"ID":"849d204d-9e90-4239-9de3-4e110b1f4554","Type":"ContainerDied","Data":"ac8c638e49e4a3e8dc6646e263440f49d010644d7d7205435cd9f621c97cf8ea"} Jul 26 15:47:32.886539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.886335 2575 scope.go:117] "RemoveContainer" containerID="bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d" Jul 26 15:47:32.896777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.896755 2575 scope.go:117] "RemoveContainer" containerID="f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1" Jul 26 15:47:32.905669 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.905649 2575 scope.go:117] "RemoveContainer" containerID="bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d" Jul 26 15:47:32.905958 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:47:32.905940 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d\": container with ID starting with bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d not found: ID does not exist" containerID="bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d" Jul 26 15:47:32.906009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.905966 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d"} err="failed to get container status \"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d\": rpc error: code = NotFound desc = could not find container \"bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d\": container with ID starting with bb3266cd7a49e6a6ed9913fe91a530b80be8d20a356f310216c2cccd49248b4d not found: ID does not exist" Jul 26 15:47:32.906009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.905983 2575 scope.go:117] "RemoveContainer" containerID="f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1" Jul 26 15:47:32.906245 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:47:32.906228 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1\": container with ID starting with f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1 not found: ID does not exist" containerID="f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1" Jul 26 15:47:32.906311 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.906250 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1"} err="failed to get container status \"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1\": rpc error: code = NotFound desc = could not find container \"f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1\": container with ID starting with f6c00c552d866888566222e3b0891853e8cecfa54219b29ce8d1cdc4503336c1 not found: ID does not exist" Jul 26 15:47:32.910553 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.910530 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:47:32.915438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:32.915417 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2ad2c2-openshift-default-668d867f76-gdkbq"] Jul 26 15:47:34.555205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:34.555172 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" path="/var/lib/kubelet/pods/849d204d-9e90-4239-9de3-4e110b1f4554/volumes" Jul 26 15:47:43.091989 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.091846 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:47:43.092496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.092109 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" podUID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" containerName="payload-pre-processing" containerID="cri-o://b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476" gracePeriod=30 Jul 26 15:47:43.360767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.360724 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:47:43.476553 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.476504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume\") pod \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " Jul 26 15:47:43.476754 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.476657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wql5j\" (UniqueName: \"kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j\") pod \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\" (UID: \"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3\") " Jul 26 15:47:43.476998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.476965 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" (UID: "c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:43.479248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.479210 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j" (OuterVolumeSpecName: "kube-api-access-wql5j") pod "c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" (UID: "c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3"). InnerVolumeSpecName "kube-api-access-wql5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:43.577605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.577563 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wql5j\" (UniqueName: \"kubernetes.io/projected/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-kube-api-access-wql5j\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:43.577605 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.577599 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:43.948641 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.948605 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" containerID="b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476" exitCode=0 Jul 26 15:47:43.948860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.948649 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" Jul 26 15:47:43.948860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.948690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" event={"ID":"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3","Type":"ContainerDied","Data":"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476"} Jul 26 15:47:43.948860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.948747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs" event={"ID":"c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3","Type":"ContainerDied","Data":"579046defd209d0f67cb802b53a14db71bbd88e90a62ec9d97bb75978431af39"} Jul 26 15:47:43.948860 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.948775 2575 scope.go:117] "RemoveContainer" containerID="b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476" Jul 26 15:47:43.958071 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.958048 2575 scope.go:117] "RemoveContainer" containerID="b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476" Jul 26 15:47:43.958364 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:47:43.958341 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476\": container with ID starting with b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476 not found: ID does not exist" containerID="b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476" Jul 26 15:47:43.958442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.958372 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476"} err="failed to get container status \"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476\": rpc error: code = NotFound desc = could not find container \"b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476\": container with ID starting with b9f44f926a42863422a2d7401649151e9ecd631bf54e79178c60980a1366c476 not found: ID does not exist" Jul 26 15:47:43.970375 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.970349 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:47:43.973203 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:43.973180 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ba5c-6699cfc668-k4dzs"] Jul 26 15:47:44.556136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:44.556100 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" path="/var/lib/kubelet/pods/c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3/volumes" Jul 26 15:47:47.838951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:47.838907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:47:53.909969 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:53.909934 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:47:53.910437 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:53.910143 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" containerID="cri-o://6c0d20a866fe0035565b5c7f324b11b1fdaf269251b47a9093c87b0332cffbb4" gracePeriod=30 Jul 26 15:47:54.782467 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.782428 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 15:47:54.782950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.782924 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" containerName="payload-pre-processing" Jul 26 15:47:54.782950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.782950 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" containerName="payload-pre-processing" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.782991 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" containerName="payload-processing" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783000 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" containerName="payload-processing" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783025 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783039 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783047 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783136 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:54.783166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783151 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48fe5d4-f7dd-4560-aadd-07a6debf4f2f" containerName="payload-processing" Jul 26 15:47:54.783600 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.783203 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d8ba00-0c6d-4a67-9ff9-5647bc2618f3" containerName="payload-pre-processing" Jul 26 15:47:54.787745 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.787725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.790312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.790289 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a20d61-openshift-default-dockercfg-mj6ws\"" Jul 26 15:47:54.796828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.796800 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 15:47:54.886204 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886504 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886670 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smd9\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886780 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.886903 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.887051 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.886904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.987556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988038 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.987993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smd9\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.988167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988438 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.988310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988533 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.988477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988576 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.988561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.988755 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.988737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.990345 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.990318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.990822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.990806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.995932 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.995909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smd9\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:54.996470 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:54.996450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token\") pod \"e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:55.100546 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:55.100463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:55.247950 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:55.247923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 15:47:55.249606 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:47:55.249576 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56664af8_cabb_42c7_8f19_c358bae60ddd.slice/crio-2f19b7f48e410bc076dd3c96719510a38e2cbe6f6e95e49ed6e6bbe108efaf01 WatchSource:0}: Error finding container 2f19b7f48e410bc076dd3c96719510a38e2cbe6f6e95e49ed6e6bbe108efaf01: Status 404 returned error can't find the container with id 2f19b7f48e410bc076dd3c96719510a38e2cbe6f6e95e49ed6e6bbe108efaf01 Jul 26 15:47:55.251938 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:55.251898 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:55.252011 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:55.251988 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:55.252047 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:55.252027 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:47:56.021188 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:56.021146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerStarted","Data":"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0"} Jul 26 15:47:56.021629 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:56.021211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerStarted","Data":"2f19b7f48e410bc076dd3c96719510a38e2cbe6f6e95e49ed6e6bbe108efaf01"} Jul 26 15:47:56.042048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:56.041998 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podStartSLOduration=2.041981481 podStartE2EDuration="2.041981481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:47:56.039808535 +0000 UTC m=+2549.937483551" watchObservedRunningTime="2026-07-26 15:47:56.041981481 +0000 UTC m=+2549.939656477" Jul 26 15:47:56.100836 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:56.100797 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:56.106197 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:56.106172 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:57.010116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:57.010069 2575 patch_prober.go:28] interesting pod/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:47:57.010320 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:57.010149 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:47:57.027165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:57.027136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:57.028205 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:57.028183 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:47:58.912862 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.912825 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 15:47:58.913345 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.913309 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="849d204d-9e90-4239-9de3-4e110b1f4554" containerName="istio-proxy" Jul 26 15:47:58.916669 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.916638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:58.919451 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.919423 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a20d61\"" Jul 26 15:47:58.919623 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.919493 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a20d61-dockercfg-5l886\"" Jul 26 15:47:58.925203 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:58.925175 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 15:47:59.027726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.027687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.027726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.027726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.042517 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.042474 2575 generic.go:356] "Generic (PLEG): container finished" podID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerID="6c0d20a866fe0035565b5c7f324b11b1fdaf269251b47a9093c87b0332cffbb4" exitCode=0 Jul 26 15:47:59.042693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.042556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerDied","Data":"6c0d20a866fe0035565b5c7f324b11b1fdaf269251b47a9093c87b0332cffbb4"} Jul 26 15:47:59.042693 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.042629 2575 scope.go:117] "RemoveContainer" containerID="4a72c0edaadb035a1528ac62fe9244fffb31d0a503b67642ced70f2d867a6da6" Jul 26 15:47:59.128823 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.128785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.129003 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.128836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.129602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.129573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.138332 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.138305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh\") pod \"payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.181227 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.181201 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:47:59.277572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.277532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.332569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.332863 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.332997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbgs\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.333410 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.333658 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.333705 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.333735 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.334909 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.335169 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.335206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.335259 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.335605 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.335993 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token\") pod \"4abc8891-3748-46f3-8cc8-37c5afa518d2\" (UID: \"4abc8891-3748-46f3-8cc8-37c5afa518d2\") " Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336668 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336688 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336704 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336719 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abc8891-3748-46f3-8cc8-37c5afa518d2-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.347312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.336734 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.352296 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.350643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:59.356559 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.352561 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs" (OuterVolumeSpecName: "kube-api-access-6cbgs") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "kube-api-access-6cbgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:59.361010 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.360528 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:47:59.363719 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.363674 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "4abc8891-3748-46f3-8cc8-37c5afa518d2" (UID: "4abc8891-3748-46f3-8cc8-37c5afa518d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:59.438116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.438086 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbgs\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-kube-api-access-6cbgs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.438116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.438115 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.438332 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.438125 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.438332 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.438133 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abc8891-3748-46f3-8cc8-37c5afa518d2-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:47:59.443854 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:47:59.443828 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 15:47:59.446347 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:47:59.446319 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5834b42a_39a0_446d_9a02_b8d18958a4c4.slice/crio-7dc6fd5198e4787bff83e4a3a17de16c984845892f7a756fbddbeaa0453a7d99 WatchSource:0}: Error finding container 7dc6fd5198e4787bff83e4a3a17de16c984845892f7a756fbddbeaa0453a7d99: Status 404 returned error can't find the container with id 7dc6fd5198e4787bff83e4a3a17de16c984845892f7a756fbddbeaa0453a7d99 Jul 26 15:48:00.049129 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.049099 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" Jul 26 15:48:00.049129 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.049107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5" event={"ID":"4abc8891-3748-46f3-8cc8-37c5afa518d2","Type":"ContainerDied","Data":"a0edee5f313913a7a03e8b6dc71a76e711b1044958bd90818c556e037d471c7e"} Jul 26 15:48:00.049696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.049171 2575 scope.go:117] "RemoveContainer" containerID="6c0d20a866fe0035565b5c7f324b11b1fdaf269251b47a9093c87b0332cffbb4" Jul 26 15:48:00.051170 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.051144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" event={"ID":"5834b42a-39a0-446d-9a02-b8d18958a4c4","Type":"ContainerStarted","Data":"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd"} Jul 26 15:48:00.051266 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.051196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" event={"ID":"5834b42a-39a0-446d-9a02-b8d18958a4c4","Type":"ContainerStarted","Data":"7dc6fd5198e4787bff83e4a3a17de16c984845892f7a756fbddbeaa0453a7d99"} Jul 26 15:48:00.051266 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.051227 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:48:00.069653 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.069587 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" podStartSLOduration=2.069570761 podStartE2EDuration="2.069570761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:00.068210858 +0000 UTC m=+2553.965885861" watchObservedRunningTime="2026-07-26 15:48:00.069570761 +0000 UTC m=+2553.967245799" Jul 26 15:48:00.084872 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.084833 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:48:00.088450 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.088419 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14ba5c-openshift-default-77d9c796c8-9f4j5"] Jul 26 15:48:00.554467 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:00.554434 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" path="/var/lib/kubelet/pods/4abc8891-3748-46f3-8cc8-37c5afa518d2/volumes" Jul 26 15:48:03.030390 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030350 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 15:48:03.030857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030838 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:03.030857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030858 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:03.030960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030879 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:03.030960 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030885 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:03.031060 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.030972 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:03.034331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.034309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.036940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.036918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-60c814-openshift-default-dockercfg-nsvm6\"" Jul 26 15:48:03.046325 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.046263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 15:48:03.074337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074567 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.074843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.074622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbssw\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.174968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbssw\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175347 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.175812 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.176064 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.175981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.176223 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.176201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.176622 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.176563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.176751 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.176626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.178556 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.178511 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.178753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.178730 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.183663 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.183640 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbssw\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.183948 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.183927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token\") pod \"e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.347212 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.347116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:03.506816 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.506783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 15:48:03.509698 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:48:03.509667 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb49da13_5eb5_4a86_8499_2181761aec33.slice/crio-f75f479efa7349b09a9b4fe0651c4598c18b527b0992531915516f0fc46c5340 WatchSource:0}: Error finding container f75f479efa7349b09a9b4fe0651c4598c18b527b0992531915516f0fc46c5340: Status 404 returned error can't find the container with id f75f479efa7349b09a9b4fe0651c4598c18b527b0992531915516f0fc46c5340 Jul 26 15:48:03.511963 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.511931 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:03.512037 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.511995 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:03.512037 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:03.512026 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:04.077025 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:04.076928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerStarted","Data":"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6"} Jul 26 15:48:04.077025 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:04.076991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerStarted","Data":"f75f479efa7349b09a9b4fe0651c4598c18b527b0992531915516f0fc46c5340"} Jul 26 15:48:04.098906 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:04.098857 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podStartSLOduration=1.098841772 podStartE2EDuration="1.098841772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:04.097497436 +0000 UTC m=+2557.995172459" watchObservedRunningTime="2026-07-26 15:48:04.098841772 +0000 UTC m=+2557.996516769" Jul 26 15:48:04.347845 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:04.347746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:04.352935 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:04.352904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:05.083734 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:05.083697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:05.084971 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:05.084952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:06.048463 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.048424 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:48:06.869668 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.869619 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 15:48:06.870131 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.870117 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abc8891-3748-46f3-8cc8-37c5afa518d2" containerName="istio-proxy" Jul 26 15:48:06.873380 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.873360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:06.875995 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.875965 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-60c814\"" Jul 26 15:48:06.876147 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.876087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-60c814-dockercfg-wcngj\"" Jul 26 15:48:06.886686 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.886654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 15:48:06.903247 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.903213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:06.903415 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:06.903255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4dz\" (UniqueName: \"kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.003878 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.003830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.004048 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.003889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4dz\" (UniqueName: \"kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.004788 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.004760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.014571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.014537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4dz\" (UniqueName: \"kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz\") pod \"payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.189377 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.189333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:07.341241 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:07.340947 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 15:48:08.109740 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:08.109696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" event={"ID":"6f4a569b-cb59-4dd1-bed8-898d03b6c891","Type":"ContainerStarted","Data":"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756"} Jul 26 15:48:08.110189 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:08.109763 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:08.110189 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:08.109785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" event={"ID":"6f4a569b-cb59-4dd1-bed8-898d03b6c891","Type":"ContainerStarted","Data":"edf8c738f4e903ed4ad5f9f39675409c222297e1327b9af67f0da206528ffd1c"} Jul 26 15:48:08.128012 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:08.127955 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" podStartSLOduration=2.127937589 podStartE2EDuration="2.127937589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:08.125465273 +0000 UTC m=+2562.023140269" watchObservedRunningTime="2026-07-26 15:48:08.127937589 +0000 UTC m=+2562.025612586" Jul 26 15:48:11.060895 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:11.060858 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 15:48:13.068497 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:13.068449 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:48:13.068947 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:13.068542 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:48:15.152705 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.152667 2575 generic.go:356] "Generic (PLEG): container finished" podID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerID="e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0" exitCode=0 Jul 26 15:48:15.153145 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.152746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerDied","Data":"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0"} Jul 26 15:48:15.153223 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.153202 2575 scope.go:117] "RemoveContainer" containerID="e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0" Jul 26 15:48:15.154312 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.154252 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:15.154425 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.154372 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:15.154488 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:15.154428 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:16.161008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:16.160976 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerStarted","Data":"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa"} Jul 26 15:48:17.100940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:17.100902 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:48:17.101350 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:17.101228 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 26 15:48:17.101550 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:17.101525 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 26 15:48:18.102025 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:18.101974 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 26 15:48:18.102568 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:18.102065 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 26 15:48:19.101658 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:19.101616 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 26 15:48:19.101965 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:19.101933 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 26 15:48:19.118054 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:19.118023 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 15:48:20.101945 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:20.101897 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 26 15:48:20.102166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:20.101977 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 26 15:48:20.696399 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:20.696359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:48:21.105121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:21.105033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:48:21.105506 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:21.105478 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:48:21.106417 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:21.106395 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 15:48:21.125496 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:21.125451 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:48:21.125669 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:21.125525 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:48:23.212832 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.212789 2575 generic.go:356] "Generic (PLEG): container finished" podID="bb49da13-5eb5-4a86-8499-2181761aec33" containerID="c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6" exitCode=0 Jul 26 15:48:23.213258 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.212847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerDied","Data":"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6"} Jul 26 15:48:23.213379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.213361 2575 scope.go:117] "RemoveContainer" containerID="c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6" Jul 26 15:48:23.214586 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.214551 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:23.214699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.214623 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:23.214699 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:23.214652 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:48:24.220297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:24.220235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerStarted","Data":"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc"} Jul 26 15:48:24.348064 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:24.348029 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:24.348406 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:24.348383 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 26 15:48:24.348470 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:24.348435 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 26 15:48:25.348166 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:25.348127 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 26 15:48:25.348686 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:25.348207 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 26 15:48:26.348560 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:26.348516 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 26 15:48:26.349046 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:26.348585 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 26 15:48:27.348444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:27.348351 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 26 15:48:27.348444 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:27.348411 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 26 15:48:28.351807 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:28.351774 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:28.352306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:28.352160 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:28.353008 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:28.352985 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 15:48:29.631381 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.631341 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:48:29.635337 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.635313 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.639933 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.639905 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"kube-root-ca.crt\"" Jul 26 15:48:29.640079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.640049 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"auth-test-model-a20d61-kserve-self-signed-certs\"" Jul 26 15:48:29.640238 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.640216 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"openshift-service-ca.crt\"" Jul 26 15:48:29.640643 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.640330 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a20d61\"/\"default-dockercfg-h8f2k\"" Jul 26 15:48:29.643377 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.643357 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:48:29.747363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.747363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.747633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.747633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.747633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdcc\" (UniqueName: \"kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.747633 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.747614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848569 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdcc\" (UniqueName: \"kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848756 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.848964 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.848765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.849157 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.849140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.849230 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.849143 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.849270 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.849227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.851265 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.851236 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.851600 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.851577 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.856409 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.856390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdcc\" (UniqueName: \"kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc\") pod \"auth-test-model-a20d61-kserve-6fc5846d67-dh9lj\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:29.947619 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:29.947581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:30.092132 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:30.092103 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:48:30.093884 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:48:30.093851 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38236e69_21dc_4250_aab1_55536630f688.slice/crio-41e13873dd549e1d2c94ac7eb5f52b05649fd80cc9a98d590565365b344572a3 WatchSource:0}: Error finding container 41e13873dd549e1d2c94ac7eb5f52b05649fd80cc9a98d590565365b344572a3: Status 404 returned error can't find the container with id 41e13873dd549e1d2c94ac7eb5f52b05649fd80cc9a98d590565365b344572a3 Jul 26 15:48:30.254165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:30.254073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" event={"ID":"38236e69-21dc-4250-aab1-55536630f688","Type":"ContainerStarted","Data":"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f"} Jul 26 15:48:30.254165 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:30.254118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" event={"ID":"38236e69-21dc-4250-aab1-55536630f688","Type":"ContainerStarted","Data":"41e13873dd549e1d2c94ac7eb5f52b05649fd80cc9a98d590565365b344572a3"} Jul 26 15:48:30.272168 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:30.272117 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" podStartSLOduration=1.272103082 podStartE2EDuration="1.272103082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:30.270146812 +0000 UTC m=+2584.167821810" watchObservedRunningTime="2026-07-26 15:48:30.272103082 +0000 UTC m=+2584.169778080" Jul 26 15:48:33.525514 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:33.525474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:48:39.948137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:39.948092 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:39.948137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:39.948148 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:39.960919 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:39.960890 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:40.321656 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:40.321621 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:48:51.739291 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:48:51.739193 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:49:17.045909 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:49:17.045860 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:49:38.335338 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:49:38.335300 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:49:56.938729 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:49:56.938691 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:50:18.527919 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:50:18.527829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:50:30.615513 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:50:30.615487 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:50:30.623288 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:50:30.623254 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:50:36.646510 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:50:36.646471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:50:58.339839 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:50:58.339797 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:51:16.634218 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:51:16.634171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:51:38.327418 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:51:38.327374 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:51:56.327880 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:51:56.327796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:52:23.833539 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:52:23.833506 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:52:41.927554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:52:41.927514 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:53:05.527939 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:05.527904 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:53:09.284353 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:09.284316 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:53:22.527668 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:22.527627 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:53:24.304366 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.304306 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" podUID="38236e69-21dc-4250-aab1-55536630f688" containerName="main" containerID="cri-o://3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f" gracePeriod=45 Jul 26 15:53:24.565006 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.564939 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:53:24.661806 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.661768 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.661973 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.661834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.661973 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.661930 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.662101 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662006 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdcc\" (UniqueName: \"kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.662101 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662033 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.662101 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662064 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs\") pod \"38236e69-21dc-4250-aab1-55536630f688\" (UID: \"38236e69-21dc-4250-aab1-55536630f688\") " Jul 26 15:53:24.662101 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662063 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:24.662379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662178 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache" (OuterVolumeSpecName: "model-cache") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:24.662379 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662337 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home" (OuterVolumeSpecName: "home") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:24.662520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662490 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-model-cache\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.662520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662508 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-home\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.662520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.662520 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-tmp-dir\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.664202 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.664182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm" (OuterVolumeSpecName: "dshm") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:24.664318 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.664292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc" (OuterVolumeSpecName: "kube-api-access-xrdcc") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "kube-api-access-xrdcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:53:24.664368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.664359 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38236e69-21dc-4250-aab1-55536630f688" (UID: "38236e69-21dc-4250-aab1-55536630f688"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:53:24.763910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.763866 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdcc\" (UniqueName: \"kubernetes.io/projected/38236e69-21dc-4250-aab1-55536630f688-kube-api-access-xrdcc\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.763910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.763905 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38236e69-21dc-4250-aab1-55536630f688-tls-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.763910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.763921 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38236e69-21dc-4250-aab1-55536630f688-dshm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:53:24.939710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.939673 2575 generic.go:356] "Generic (PLEG): container finished" podID="38236e69-21dc-4250-aab1-55536630f688" containerID="3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f" exitCode=0 Jul 26 15:53:24.939879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.939720 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" Jul 26 15:53:24.939879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.939765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" event={"ID":"38236e69-21dc-4250-aab1-55536630f688","Type":"ContainerDied","Data":"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f"} Jul 26 15:53:24.939879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.939818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj" event={"ID":"38236e69-21dc-4250-aab1-55536630f688","Type":"ContainerDied","Data":"41e13873dd549e1d2c94ac7eb5f52b05649fd80cc9a98d590565365b344572a3"} Jul 26 15:53:24.939879 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.939838 2575 scope.go:117] "RemoveContainer" containerID="3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f" Jul 26 15:53:24.949768 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.949749 2575 scope.go:117] "RemoveContainer" containerID="3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f" Jul 26 15:53:24.950030 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:53:24.950014 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f\": container with ID starting with 3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f not found: ID does not exist" containerID="3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f" Jul 26 15:53:24.950075 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.950038 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f"} err="failed to get container status \"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f\": rpc error: code = NotFound desc = could not find container \"3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f\": container with ID starting with 3f103b78dadf510ced52c813e0c55e19de66fb6dd6c11eef4f1b8249d4cc508f not found: ID does not exist" Jul 26 15:53:24.962133 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.962108 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:53:24.967306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:24.967284 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a20d61/auth-test-model-a20d61-kserve-6fc5846d67-dh9lj"] Jul 26 15:53:26.559297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:26.556633 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38236e69-21dc-4250-aab1-55536630f688" path="/var/lib/kubelet/pods/38236e69-21dc-4250-aab1-55536630f688/volumes" Jul 26 15:53:42.830594 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:53:42.830560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:54:24.033080 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.033044 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:54:24.033552 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.033541 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38236e69-21dc-4250-aab1-55536630f688" containerName="main" Jul 26 15:54:24.033603 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.033554 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="38236e69-21dc-4250-aab1-55536630f688" containerName="main" Jul 26 15:54:24.033642 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.033626 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="38236e69-21dc-4250-aab1-55536630f688" containerName="main" Jul 26 15:54:24.036873 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.036855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.039484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.039465 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"kube-root-ca.crt\"" Jul 26 15:54:24.040562 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.040544 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"openshift-service-ca.crt\"" Jul 26 15:54:24.040937 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.040918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"default-dockercfg-vkltq\"" Jul 26 15:54:24.041057 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.040941 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"sub-test-model-60c814-kserve-self-signed-certs\"" Jul 26 15:54:24.045638 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.045611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:54:24.120728 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.120898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.120898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.120898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.120898 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.121074 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.120935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qbs\" (UniqueName: \"kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.221833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.221833 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qbs\" (UniqueName: \"kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.221974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222327 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.222298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222398 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.222338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.222398 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.222350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.224413 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.224388 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.224516 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.224503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.229695 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.229671 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qbs\" (UniqueName: \"kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs\") pod \"sub-test-model-60c814-kserve-5895f5689f-cpt5f\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.347507 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.347416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:24.495201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:24.495167 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:54:24.496631 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:54:24.496582 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d2082ce_5999_4b5d_9f10_8fed5436ce35.slice/crio-195e0ec3d51b01a13661a592466893859201189cf3e002c32aca8a5fbbc5e359 WatchSource:0}: Error finding container 195e0ec3d51b01a13661a592466893859201189cf3e002c32aca8a5fbbc5e359: Status 404 returned error can't find the container with id 195e0ec3d51b01a13661a592466893859201189cf3e002c32aca8a5fbbc5e359 Jul 26 15:54:25.278901 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:25.278856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" event={"ID":"6d2082ce-5999-4b5d-9f10-8fed5436ce35","Type":"ContainerStarted","Data":"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1"} Jul 26 15:54:25.279331 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:25.278912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" event={"ID":"6d2082ce-5999-4b5d-9f10-8fed5436ce35","Type":"ContainerStarted","Data":"195e0ec3d51b01a13661a592466893859201189cf3e002c32aca8a5fbbc5e359"} Jul 26 15:54:25.297731 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:25.297679 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" podStartSLOduration=1.297661607 podStartE2EDuration="1.297661607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:54:25.294651468 +0000 UTC m=+2939.192326477" watchObservedRunningTime="2026-07-26 15:54:25.297661607 +0000 UTC m=+2939.195336603" Jul 26 15:54:28.032520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:28.032474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:54:34.348326 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:34.348255 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:34.348791 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:34.348350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:34.361056 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:34.361025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:35.345710 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:35.345677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:54:59.535222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:54:59.535131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:55:17.729441 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:17.729402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:55:30.654762 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:30.654735 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:55:30.663815 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:30.663782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 15:55:30.819164 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:30.819126 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:55:32.634655 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:32.634621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:55:45.839375 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:45.839311 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" podUID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" containerName="main" containerID="cri-o://a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1" gracePeriod=45 Jul 26 15:55:46.099057 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.098990 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:55:46.210535 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210497 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.210715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210561 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.210715 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210588 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.210840 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210766 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.210904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210836 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.210904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210863 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home" (OuterVolumeSpecName: "home") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:46.210904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.210868 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95qbs\" (UniqueName: \"kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs\") pod \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\" (UID: \"6d2082ce-5999-4b5d-9f10-8fed5436ce35\") " Jul 26 15:55:46.211091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.211040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:46.211222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.211099 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:46.211364 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.211347 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-home\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.211421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.211368 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tmp-dir\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.211421 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.211380 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-model-cache\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.213170 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.213148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs" (OuterVolumeSpecName: "kube-api-access-95qbs") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "kube-api-access-95qbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:55:46.213302 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.213268 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm" (OuterVolumeSpecName: "dshm") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:55:46.213363 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.213301 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d2082ce-5999-4b5d-9f10-8fed5436ce35" (UID: "6d2082ce-5999-4b5d-9f10-8fed5436ce35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:55:46.312750 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.312705 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qbs\" (UniqueName: \"kubernetes.io/projected/6d2082ce-5999-4b5d-9f10-8fed5436ce35-kube-api-access-95qbs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.312750 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.312742 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d2082ce-5999-4b5d-9f10-8fed5436ce35-tls-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.312750 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.312754 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d2082ce-5999-4b5d-9f10-8fed5436ce35-dshm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:55:46.746843 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.746804 2575 generic.go:356] "Generic (PLEG): container finished" podID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" containerID="a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1" exitCode=0 Jul 26 15:55:46.747022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.746854 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" Jul 26 15:55:46.747022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.746896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" event={"ID":"6d2082ce-5999-4b5d-9f10-8fed5436ce35","Type":"ContainerDied","Data":"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1"} Jul 26 15:55:46.747022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.746944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f" event={"ID":"6d2082ce-5999-4b5d-9f10-8fed5436ce35","Type":"ContainerDied","Data":"195e0ec3d51b01a13661a592466893859201189cf3e002c32aca8a5fbbc5e359"} Jul 26 15:55:46.747022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.746967 2575 scope.go:117] "RemoveContainer" containerID="a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1" Jul 26 15:55:46.756303 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.756263 2575 scope.go:117] "RemoveContainer" containerID="a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1" Jul 26 15:55:46.756571 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:55:46.756550 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1\": container with ID starting with a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1 not found: ID does not exist" containerID="a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1" Jul 26 15:55:46.756634 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.756583 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1"} err="failed to get container status \"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1\": rpc error: code = NotFound desc = could not find container \"a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1\": container with ID starting with a4ad9f1787649a5f188dc10b10fbc87a4a47da06c4b813f4b4f31d3929d8deb1 not found: ID does not exist" Jul 26 15:55:46.764231 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.764206 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:55:46.767434 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:46.767411 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/sub-test-model-60c814-kserve-5895f5689f-cpt5f"] Jul 26 15:55:47.036857 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:47.036822 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:55:48.554766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:48.554730 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" path="/var/lib/kubelet/pods/6d2082ce-5999-4b5d-9f10-8fed5436ce35/volumes" Jul 26 15:55:57.834000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:55:57.833966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:56:16.038239 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.038200 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:56:16.038767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.038745 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" containerName="main" Jul 26 15:56:16.038767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.038765 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" containerName="main" Jul 26 15:56:16.038859 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.038845 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2082ce-5999-4b5d-9f10-8fed5436ce35" containerName="main" Jul 26 15:56:16.043040 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.043013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.050009 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.049982 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"rate-test-model-60c814-kserve-self-signed-certs\"" Jul 26 15:56:16.050172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.050011 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"kube-root-ca.crt\"" Jul 26 15:56:16.050172 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.049982 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"default-dockercfg-vkltq\"" Jul 26 15:56:16.050979 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.050957 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60c814\"/\"openshift-service-ca.crt\"" Jul 26 15:56:16.054911 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.054887 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:56:16.094558 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.094748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.094748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578n9\" (UniqueName: \"kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.094748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.094883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.094883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.094786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196116 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578n9\" (UniqueName: \"kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196339 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196787 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196886 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.196886 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.196859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.198736 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.198715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.198913 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.198894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.212590 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.212564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-578n9\" (UniqueName: \"kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9\") pod \"rate-test-model-60c814-kserve-6749f9d57f-4wpcx\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.357230 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.357142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:16.499901 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.499872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:56:16.501471 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:56:16.501441 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f3f474b_c68c_4427_b469_93abb46f645a.slice/crio-dafc80baf266d19df538fca54e83671259a697212b819a96d7347823c51c7e62 WatchSource:0}: Error finding container dafc80baf266d19df538fca54e83671259a697212b819a96d7347823c51c7e62: Status 404 returned error can't find the container with id dafc80baf266d19df538fca54e83671259a697212b819a96d7347823c51c7e62 Jul 26 15:56:16.923822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.923724 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" event={"ID":"1f3f474b-c68c-4427-b469-93abb46f645a","Type":"ContainerStarted","Data":"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b"} Jul 26 15:56:16.923822 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.923770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" event={"ID":"1f3f474b-c68c-4427-b469-93abb46f645a","Type":"ContainerStarted","Data":"dafc80baf266d19df538fca54e83671259a697212b819a96d7347823c51c7e62"} Jul 26 15:56:16.941228 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:16.941172 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" podStartSLOduration=0.941154657 podStartE2EDuration="941.154657ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:56:16.940246508 +0000 UTC m=+3050.837921525" watchObservedRunningTime="2026-07-26 15:56:16.941154657 +0000 UTC m=+3050.838829654" Jul 26 15:56:20.233108 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:20.233074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:56:26.357704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:26.357660 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:26.357704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:26.357712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:26.371083 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:26.371052 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:56:26.992442 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:56:26.992401 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:57:04.731516 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:04.731478 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:57:24.231074 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:24.231035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:57:32.020476 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:32.020437 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:57:39.297726 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.297690 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:39.301284 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.301254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:39.303767 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.303743 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fxpw2\"" Jul 26 15:57:39.307299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.307250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:39.367073 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.367032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9gk\" (UniqueName: \"kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk\") pod \"test-curl-7535-140214313752448\" (UID: \"eac6e935-80a7-4808-9290-9899a4c485dc\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:39.432743 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.432706 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:57:39.467597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.467554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9gk\" (UniqueName: \"kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk\") pod \"test-curl-7535-140214313752448\" (UID: \"eac6e935-80a7-4808-9290-9899a4c485dc\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:39.475477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.475450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9gk\" (UniqueName: \"kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk\") pod \"test-curl-7535-140214313752448\" (UID: \"eac6e935-80a7-4808-9290-9899a4c485dc\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:39.612803 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.612687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:39.753441 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.753267 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:39.755911 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:57:39.755876 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac6e935_80a7_4808_9290_9899a4c485dc.slice/crio-97d684d43317c1fb6066c0b014225eafdd972a27f52a2fffa53baa72cf1c5b37 WatchSource:0}: Error finding container 97d684d43317c1fb6066c0b014225eafdd972a27f52a2fffa53baa72cf1c5b37: Status 404 returned error can't find the container with id 97d684d43317c1fb6066c0b014225eafdd972a27f52a2fffa53baa72cf1c5b37 Jul 26 15:57:39.757612 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:39.757592 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:57:40.402798 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:40.402760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214313752448" event={"ID":"eac6e935-80a7-4808-9290-9899a4c485dc","Type":"ContainerStarted","Data":"97d684d43317c1fb6066c0b014225eafdd972a27f52a2fffa53baa72cf1c5b37"} Jul 26 15:57:41.410174 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:41.410135 2575 generic.go:356] "Generic (PLEG): container finished" podID="eac6e935-80a7-4808-9290-9899a4c485dc" containerID="6f7159dc9389c71338afb80efab4b3e520b998fe736912f6cc53f315c3767590" exitCode=0 Jul 26 15:57:41.410583 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:41.410224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214313752448" event={"ID":"eac6e935-80a7-4808-9290-9899a4c485dc","Type":"ContainerDied","Data":"6f7159dc9389c71338afb80efab4b3e520b998fe736912f6cc53f315c3767590"} Jul 26 15:57:42.558406 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.558376 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:42.592789 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.592753 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214313752448_eac6e935-80a7-4808-9290-9899a4c485dc/test-curl-7535-140214313752448/0.log" Jul 26 15:57:42.598940 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.598917 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw9gk\" (UniqueName: \"kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk\") pod \"eac6e935-80a7-4808-9290-9899a4c485dc\" (UID: \"eac6e935-80a7-4808-9290-9899a4c485dc\") " Jul 26 15:57:42.601219 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.601184 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk" (OuterVolumeSpecName: "kube-api-access-nw9gk") pod "eac6e935-80a7-4808-9290-9899a4c485dc" (UID: "eac6e935-80a7-4808-9290-9899a4c485dc"). InnerVolumeSpecName "kube-api-access-nw9gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:42.615505 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.615478 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:42.619225 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.619199 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:42.700022 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.699987 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9gk\" (UniqueName: \"kubernetes.io/projected/eac6e935-80a7-4808-9290-9899a4c485dc-kube-api-access-nw9gk\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:42.738223 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.738177 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:42.738696 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.738675 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac6e935-80a7-4808-9290-9899a4c485dc" containerName="test-curl-7535-140214313752448" Jul 26 15:57:42.738811 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.738698 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac6e935-80a7-4808-9290-9899a4c485dc" containerName="test-curl-7535-140214313752448" Jul 26 15:57:42.738877 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.738839 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac6e935-80a7-4808-9290-9899a4c485dc" containerName="test-curl-7535-140214313752448" Jul 26 15:57:42.741997 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.741967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:42.747611 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.747586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:42.800670 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.800633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9w92\" (UniqueName: \"kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92\") pod \"test-curl-7535-140214313752448\" (UID: \"7d0779ae-f17e-4475-b663-b21eb41d0c3b\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:42.902036 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.901985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9w92\" (UniqueName: \"kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92\") pod \"test-curl-7535-140214313752448\" (UID: \"7d0779ae-f17e-4475-b663-b21eb41d0c3b\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:42.910392 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:42.910363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9w92\" (UniqueName: \"kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92\") pod \"test-curl-7535-140214313752448\" (UID: \"7d0779ae-f17e-4475-b663-b21eb41d0c3b\") " pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:43.053689 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.053586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:43.200368 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.200335 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:43.200874 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:57:43.200847 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d0779ae_f17e_4475_b663_b21eb41d0c3b.slice/crio-d35eef8cdd8032a714dc12703362c4545835da34aadcbe85c4b684f8cf8bf8b5 WatchSource:0}: Error finding container d35eef8cdd8032a714dc12703362c4545835da34aadcbe85c4b684f8cf8bf8b5: Status 404 returned error can't find the container with id d35eef8cdd8032a714dc12703362c4545835da34aadcbe85c4b684f8cf8bf8b5 Jul 26 15:57:43.422727 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.422634 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:43.422727 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.422648 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d684d43317c1fb6066c0b014225eafdd972a27f52a2fffa53baa72cf1c5b37" Jul 26 15:57:43.423910 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.423881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214313752448" event={"ID":"7d0779ae-f17e-4475-b663-b21eb41d0c3b","Type":"ContainerStarted","Data":"d35eef8cdd8032a714dc12703362c4545835da34aadcbe85c4b684f8cf8bf8b5"} Jul 26 15:57:43.434749 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:43.434709 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7535-140214313752448" oldPodUID="eac6e935-80a7-4808-9290-9899a4c485dc" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" Jul 26 15:57:44.431050 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:44.431012 2575 generic.go:356] "Generic (PLEG): container finished" podID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" containerID="34ff8170030a3174cd6f353d1a4a4e99e02e8f817208dd986828cb2b99aa048e" exitCode=0 Jul 26 15:57:44.431494 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:44.431104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214313752448" event={"ID":"7d0779ae-f17e-4475-b663-b21eb41d0c3b","Type":"ContainerDied","Data":"34ff8170030a3174cd6f353d1a4a4e99e02e8f817208dd986828cb2b99aa048e"} Jul 26 15:57:44.555521 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:44.555485 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac6e935-80a7-4808-9290-9899a4c485dc" path="/var/lib/kubelet/pods/eac6e935-80a7-4808-9290-9899a4c485dc/volumes" Jul 26 15:57:45.574572 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.574546 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:45.593827 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.593799 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214313752448_7d0779ae-f17e-4475-b663-b21eb41d0c3b/test-curl-7535-140214313752448/0.log" Jul 26 15:57:45.619491 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.619452 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:45.622869 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.622840 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214313752448"] Jul 26 15:57:45.627709 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.627685 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9w92\" (UniqueName: \"kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92\") pod \"7d0779ae-f17e-4475-b663-b21eb41d0c3b\" (UID: \"7d0779ae-f17e-4475-b663-b21eb41d0c3b\") " Jul 26 15:57:45.629983 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.629950 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92" (OuterVolumeSpecName: "kube-api-access-m9w92") pod "7d0779ae-f17e-4475-b663-b21eb41d0c3b" (UID: "7d0779ae-f17e-4475-b663-b21eb41d0c3b"). InnerVolumeSpecName "kube-api-access-m9w92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:45.728878 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:45.728791 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9w92\" (UniqueName: \"kubernetes.io/projected/7d0779ae-f17e-4475-b663-b21eb41d0c3b-kube-api-access-m9w92\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:46.443624 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:46.443591 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:46.443624 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:46.443616 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35eef8cdd8032a714dc12703362c4545835da34aadcbe85c4b684f8cf8bf8b5" Jul 26 15:57:46.453409 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:57:46.453382 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214313752448\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:46.553348 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:57:46.553303 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214313752448\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" pod="opendatahub/test-curl-7535-140214313752448" Jul 26 15:57:46.554587 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:46.554566 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" path="/var/lib/kubelet/pods/7d0779ae-f17e-4475-b663-b21eb41d0c3b/volumes" Jul 26 15:57:47.040210 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.040160 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" podUID="1f3f474b-c68c-4427-b469-93abb46f645a" containerName="main" containerID="cri-o://9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b" gracePeriod=45 Jul 26 15:57:47.311013 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.310986 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:57:47.345601 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345571 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.345772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345631 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.345772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345649 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.345772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345691 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-578n9\" (UniqueName: \"kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.345772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345749 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.346002 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345793 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir\") pod \"1f3f474b-c68c-4427-b469-93abb46f645a\" (UID: \"1f3f474b-c68c-4427-b469-93abb46f645a\") " Jul 26 15:57:47.346002 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345942 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:47.346104 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.345992 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home" (OuterVolumeSpecName: "home") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:47.346155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.346124 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-model-cache\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.346155 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.346143 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-home\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.346301 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.346164 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:47.348088 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.348065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm" (OuterVolumeSpecName: "dshm") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:47.348477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.348457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:57:47.348477 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.348462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9" (OuterVolumeSpecName: "kube-api-access-578n9") pod "1f3f474b-c68c-4427-b469-93abb46f645a" (UID: "1f3f474b-c68c-4427-b469-93abb46f645a"). InnerVolumeSpecName "kube-api-access-578n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:47.446625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.446594 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-dshm\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.446625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.446620 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3f474b-c68c-4427-b469-93abb46f645a-tmp-dir\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.446625 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.446628 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3f474b-c68c-4427-b469-93abb46f645a-tls-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.446883 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.446638 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-578n9\" (UniqueName: \"kubernetes.io/projected/1f3f474b-c68c-4427-b469-93abb46f645a-kube-api-access-578n9\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:47.449899 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.449871 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f3f474b-c68c-4427-b469-93abb46f645a" containerID="9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b" exitCode=0 Jul 26 15:57:47.449994 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.449917 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" Jul 26 15:57:47.449994 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.449959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" event={"ID":"1f3f474b-c68c-4427-b469-93abb46f645a","Type":"ContainerDied","Data":"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b"} Jul 26 15:57:47.450554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.450007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx" event={"ID":"1f3f474b-c68c-4427-b469-93abb46f645a","Type":"ContainerDied","Data":"dafc80baf266d19df538fca54e83671259a697212b819a96d7347823c51c7e62"} Jul 26 15:57:47.450554 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.450029 2575 scope.go:117] "RemoveContainer" containerID="9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b" Jul 26 15:57:47.459678 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.459658 2575 scope.go:117] "RemoveContainer" containerID="9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b" Jul 26 15:57:47.459945 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:57:47.459926 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b\": container with ID starting with 9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b not found: ID does not exist" containerID="9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b" Jul 26 15:57:47.459996 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.459954 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b"} err="failed to get container status \"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b\": rpc error: code = NotFound desc = could not find container \"9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b\": container with ID starting with 9e0c8b3bbeb711578bf12720e7ed98d7be9d7e1221386e2caa8b18d1d2a3530b not found: ID does not exist" Jul 26 15:57:47.472083 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.472030 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:57:47.475190 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:47.475166 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60c814/rate-test-model-60c814-kserve-6749f9d57f-4wpcx"] Jul 26 15:57:48.321307 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321262 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:48.321704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321692 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" containerName="test-curl-7535-140214313752448" Jul 26 15:57:48.321704 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321704 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" containerName="test-curl-7535-140214313752448" Jul 26 15:57:48.321778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321721 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3f474b-c68c-4427-b469-93abb46f645a" containerName="main" Jul 26 15:57:48.321778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321727 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3f474b-c68c-4427-b469-93abb46f645a" containerName="main" Jul 26 15:57:48.321845 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321787 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0779ae-f17e-4475-b663-b21eb41d0c3b" containerName="test-curl-7535-140214313752448" Jul 26 15:57:48.321845 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.321799 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3f474b-c68c-4427-b469-93abb46f645a" containerName="main" Jul 26 15:57:48.326033 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.326001 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.328506 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.328485 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fxpw2\"" Jul 26 15:57:48.328872 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.328849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:48.455645 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.455614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f894\" (UniqueName: \"kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894\") pod \"test-curl-7535-140214310254176\" (UID: \"bfd23067-7e88-47e6-9831-87e12112d841\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.554299 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.554254 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f3f474b-c68c-4427-b469-93abb46f645a" path="/var/lib/kubelet/pods/1f3f474b-c68c-4427-b469-93abb46f645a/volumes" Jul 26 15:57:48.556790 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.556756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f894\" (UniqueName: \"kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894\") pod \"test-curl-7535-140214310254176\" (UID: \"bfd23067-7e88-47e6-9831-87e12112d841\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.564753 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.564727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f894\" (UniqueName: \"kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894\") pod \"test-curl-7535-140214310254176\" (UID: \"bfd23067-7e88-47e6-9831-87e12112d841\") " pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.636103 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.636004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:48.777959 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:48.777927 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:48.779923 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:57:48.779893 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfd23067_7e88_47e6_9831_87e12112d841.slice/crio-9554327881dcc330537ea0d875bf1727eefff0a80ac34d05dc79c7f4dbac5525 WatchSource:0}: Error finding container 9554327881dcc330537ea0d875bf1727eefff0a80ac34d05dc79c7f4dbac5525: Status 404 returned error can't find the container with id 9554327881dcc330537ea0d875bf1727eefff0a80ac34d05dc79c7f4dbac5525 Jul 26 15:57:49.463793 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:49.463758 2575 generic.go:356] "Generic (PLEG): container finished" podID="bfd23067-7e88-47e6-9831-87e12112d841" containerID="c7c67d4e5d3ce69b21cd32be19bafc655ceed99236958542b9dd8b0d509bc75b" exitCode=0 Jul 26 15:57:49.464212 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:49.463842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310254176" event={"ID":"bfd23067-7e88-47e6-9831-87e12112d841","Type":"ContainerDied","Data":"c7c67d4e5d3ce69b21cd32be19bafc655ceed99236958542b9dd8b0d509bc75b"} Jul 26 15:57:49.464212 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:49.463881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7535-140214310254176" event={"ID":"bfd23067-7e88-47e6-9831-87e12112d841","Type":"ContainerStarted","Data":"9554327881dcc330537ea0d875bf1727eefff0a80ac34d05dc79c7f4dbac5525"} Jul 26 15:57:50.631907 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.631876 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:50.650958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.650918 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7535-140214310254176_bfd23067-7e88-47e6-9831-87e12112d841/test-curl-7535-140214310254176/0.log" Jul 26 15:57:50.673998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.673953 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:50.676852 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.676819 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f894\" (UniqueName: \"kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894\") pod \"bfd23067-7e88-47e6-9831-87e12112d841\" (UID: \"bfd23067-7e88-47e6-9831-87e12112d841\") " Jul 26 15:57:50.679397 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.679359 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894" (OuterVolumeSpecName: "kube-api-access-2f894") pod "bfd23067-7e88-47e6-9831-87e12112d841" (UID: "bfd23067-7e88-47e6-9831-87e12112d841"). InnerVolumeSpecName "kube-api-access-2f894". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:50.679904 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.679881 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7535-140214310254176"] Jul 26 15:57:50.778091 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:50.777992 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f894\" (UniqueName: \"kubernetes.io/projected/bfd23067-7e88-47e6-9831-87e12112d841-kube-api-access-2f894\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 15:57:51.434133 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.434097 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 15:57:51.434593 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.434576 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd23067-7e88-47e6-9831-87e12112d841" containerName="test-curl-7535-140214310254176" Jul 26 15:57:51.434652 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.434595 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd23067-7e88-47e6-9831-87e12112d841" containerName="test-curl-7535-140214310254176" Jul 26 15:57:51.434712 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.434702 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd23067-7e88-47e6-9831-87e12112d841" containerName="test-curl-7535-140214310254176" Jul 26 15:57:51.438060 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.438038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.440565 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.440544 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d57e1d-openshift-default-dockercfg-2mqp4\"" Jul 26 15:57:51.449317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.449292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 15:57:51.476573 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.476542 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:51.476766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.476572 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9554327881dcc330537ea0d875bf1727eefff0a80ac34d05dc79c7f4dbac5525" Jul 26 15:57:51.485765 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.485858 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.485858 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwln\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.485858 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.486020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.486020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.486020 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.485992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.486177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.486025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.486177 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.486053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.488594 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:57:51.488565 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310254176\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="bfd23067-7e88-47e6-9831-87e12112d841" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:51.587349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwln\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587551 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.587881 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.588136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.587948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.588136 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.588075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.588244 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.588160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.588508 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.588451 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.590147 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.590121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.590340 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.590319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.596156 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.596127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.596243 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.596174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwln\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln\") pod \"e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.750778 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.750674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:51.898201 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.898169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 15:57:51.899131 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:57:51.899100 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87efd397_f344_4908_b3ad_47cd1c406759.slice/crio-a375fb7d2ddf692ebdd8aa56ec3a0c485d565cf7324ca9fec19d3bece401a1e6 WatchSource:0}: Error finding container a375fb7d2ddf692ebdd8aa56ec3a0c485d565cf7324ca9fec19d3bece401a1e6: Status 404 returned error can't find the container with id a375fb7d2ddf692ebdd8aa56ec3a0c485d565cf7324ca9fec19d3bece401a1e6 Jul 26 15:57:51.901810 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.901779 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:57:51.901931 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.901845 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:57:51.901931 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:51.901874 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:57:52.485249 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.485206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerStarted","Data":"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3"} Jul 26 15:57:52.485457 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.485294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerStarted","Data":"a375fb7d2ddf692ebdd8aa56ec3a0c485d565cf7324ca9fec19d3bece401a1e6"} Jul 26 15:57:52.487662 ip-10-0-139-15 kubenswrapper[2575]: E0726 15:57:52.487621 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7535-140214310254176\" is forbidden: User \"system:node:ip-10-0-139-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-15.ec2.internal' and this object" podUID="bfd23067-7e88-47e6-9831-87e12112d841" pod="opendatahub/test-curl-7535-140214310254176" Jul 26 15:57:52.508423 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.508377 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podStartSLOduration=1.508362552 podStartE2EDuration="1.508362552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:52.506554553 +0000 UTC m=+3146.404229748" watchObservedRunningTime="2026-07-26 15:57:52.508362552 +0000 UTC m=+3146.406037549" Jul 26 15:57:52.555125 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.555089 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd23067-7e88-47e6-9831-87e12112d841" path="/var/lib/kubelet/pods/bfd23067-7e88-47e6-9831-87e12112d841/volumes" Jul 26 15:57:52.751597 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.751489 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:52.756521 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:52.756489 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:53.494317 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:53.494263 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:53.495410 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:53.495385 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:57:55.258817 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.258778 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 15:57:55.262828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.262804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.265998 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.265971 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d57e1d\"" Jul 26 15:57:55.266472 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.266456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d57e1d-dockercfg-8zw6h\"" Jul 26 15:57:55.274682 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.274654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 15:57:55.326644 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.326604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgzg\" (UniqueName: \"kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.326891 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.326700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.427809 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.427766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgzg\" (UniqueName: \"kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.427993 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.427844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.428604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.428582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.435909 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.435877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgzg\" (UniqueName: \"kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg\") pod \"payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.574349 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.574235 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:55.723741 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:55.723712 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 15:57:55.725210 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:57:55.725171 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df9add7_5ee4_42c9_8e1a_15374dfedb17.slice/crio-1c5a125515c3055cd144ddc36ad64b58b0567215acfe46ff643d2bce8488b4ac WatchSource:0}: Error finding container 1c5a125515c3055cd144ddc36ad64b58b0567215acfe46ff643d2bce8488b4ac: Status 404 returned error can't find the container with id 1c5a125515c3055cd144ddc36ad64b58b0567215acfe46ff643d2bce8488b4ac Jul 26 15:57:56.516006 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:56.515964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" event={"ID":"4df9add7-5ee4-42c9-8e1a-15374dfedb17","Type":"ContainerStarted","Data":"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2"} Jul 26 15:57:56.516474 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:56.516017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" event={"ID":"4df9add7-5ee4-42c9-8e1a-15374dfedb17","Type":"ContainerStarted","Data":"1c5a125515c3055cd144ddc36ad64b58b0567215acfe46ff643d2bce8488b4ac"} Jul 26 15:57:56.516474 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:56.516107 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:57:56.533115 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:57:56.533058 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" podStartSLOduration=1.533042584 podStartE2EDuration="1.533042584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:57:56.531372682 +0000 UTC m=+3150.429047678" watchObservedRunningTime="2026-07-26 15:57:56.533042584 +0000 UTC m=+3150.430717632" Jul 26 15:58:03.635079 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:03.635040 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:58:07.523687 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:07.523656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 15:58:10.086118 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.086080 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 15:58:10.090330 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.090302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.092946 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.092898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cb6b99-openshift-default-dockercfg-gmq98\"" Jul 26 15:58:10.104131 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.104099 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 15:58:10.170827 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.170786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.170831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.170949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171004 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171141 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171141 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171141 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nz9g\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171141 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.171248 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.171219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.271709 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.271668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.271924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.271716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nz9g\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.271924 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.271849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.271983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272169 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272169 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272297 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272462 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272520 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272602 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.272647 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.272606 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.274828 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.274802 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.274973 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.274910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.281773 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.281743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nz9g\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.282242 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.282216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token\") pod \"e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.403901 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.403810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:10.563167 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.563139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 15:58:10.564885 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:58:10.564850 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bc15aa_02e5_4838_bc86_522eb72c44c4.slice/crio-7e4bd60e25ff37618793cfbe6b70cd255e5de02834f5edb0abcddae0c0436a8b WatchSource:0}: Error finding container 7e4bd60e25ff37618793cfbe6b70cd255e5de02834f5edb0abcddae0c0436a8b: Status 404 returned error can't find the container with id 7e4bd60e25ff37618793cfbe6b70cd255e5de02834f5edb0abcddae0c0436a8b Jul 26 15:58:10.567261 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.567229 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:10.567389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.567335 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:10.567389 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.567368 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:10.620159 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:10.620125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerStarted","Data":"7e4bd60e25ff37618793cfbe6b70cd255e5de02834f5edb0abcddae0c0436a8b"} Jul 26 15:58:11.627723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.627680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerStarted","Data":"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741"} Jul 26 15:58:11.629320 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.629263 2575 generic.go:356] "Generic (PLEG): container finished" podID="87efd397-f344-4908-b3ad-47cd1c406759" containerID="f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3" exitCode=0 Jul 26 15:58:11.629456 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.629301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerDied","Data":"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3"} Jul 26 15:58:11.629706 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.629693 2575 scope.go:117] "RemoveContainer" containerID="f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3" Jul 26 15:58:11.630712 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.630684 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:11.630777 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.630762 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:11.630817 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.630800 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:11.649772 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:11.649678 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podStartSLOduration=1.649664134 podStartE2EDuration="1.649664134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:11.647492678 +0000 UTC m=+3165.545167675" watchObservedRunningTime="2026-07-26 15:58:11.649664134 +0000 UTC m=+3165.547339130" Jul 26 15:58:12.404475 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.404436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:12.409493 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.409461 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:12.637189 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.637150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerStarted","Data":"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35"} Jul 26 15:58:12.637685 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.637408 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:12.638384 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.638362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:12.751607 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.751570 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:58:12.751835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.751811 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 26 15:58:12.751930 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:12.751868 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 26 15:58:13.752086 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:13.752045 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 26 15:58:13.752692 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:13.752137 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 26 15:58:14.751356 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:14.751318 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 26 15:58:14.751549 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:14.751374 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 26 15:58:15.751577 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:15.751538 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 26 15:58:15.752058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:15.751605 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 26 15:58:16.755023 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:16.754990 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:58:16.755450 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:16.755310 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:58:16.756061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:16.756042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 15:58:24.534388 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:24.534347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:58:32.769440 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.769405 2575 generic.go:356] "Generic (PLEG): container finished" podID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerID="ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741" exitCode=0 Jul 26 15:58:32.769862 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.769480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerDied","Data":"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741"} Jul 26 15:58:32.769917 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.769901 2575 scope.go:117] "RemoveContainer" containerID="ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741" Jul 26 15:58:32.771061 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.771032 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:32.771121 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.771101 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:32.771160 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:32.771135 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 26 15:58:33.777460 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:33.777413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerStarted","Data":"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469"} Jul 26 15:58:34.404548 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:34.404503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:34.404769 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:34.404749 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Jul 26 15:58:34.404850 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:34.404800 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Jul 26 15:58:35.404571 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:35.404537 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Jul 26 15:58:35.405014 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:35.404610 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Jul 26 15:58:36.405058 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:36.405000 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Jul 26 15:58:36.405604 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:36.405085 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Jul 26 15:58:37.405137 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:37.405098 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Jul 26 15:58:37.405837 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:37.405788 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Jul 26 15:58:38.408106 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:38.408069 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:38.408617 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:38.408483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:38.409222 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:38.409204 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 15:58:57.139508 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:58:57.139464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:59:25.334306 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:25.334252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:59:35.358251 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:35.358221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8_4df9add7-5ee4-42c9-8e1a-15374dfedb17/payload-processing/0.log" Jul 26 15:59:39.790204 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.790139 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 15:59:39.794089 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.794065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:39.796766 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.796743 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-35252a\"" Jul 26 15:59:39.806194 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.806165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 15:59:39.813206 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.813178 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 15:59:39.816958 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.816934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:39.819608 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.819583 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-35252a-dockercfg-t8wgb\"" Jul 26 15:59:39.827320 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.827257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 15:59:39.899884 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.899846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khcw\" (UniqueName: \"kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:39.900081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.899889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:39.900081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.899954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gff74\" (UniqueName: \"kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:39.900081 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:39.900007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.001555 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.001520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.001748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.001600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khcw\" (UniqueName: \"kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.001748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.001629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:40.001748 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.001657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gff74\" (UniqueName: \"kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:40.002235 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.002210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.002484 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.002466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:40.009690 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.009660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khcw\" (UniqueName: \"kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw\") pod \"payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.009834 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.009814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gff74\" (UniqueName: \"kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74\") pod \"payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:40.105951 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.105855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:40.129383 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.129340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:40.265358 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.265307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 15:59:40.267650 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:59:40.267621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbdb079_71e3_44b8_b33a_62d965f05dfe.slice/crio-dbc27a8e987f01469e91fcc29387939652d2da779c28e3bed8b02821b48164d4 WatchSource:0}: Error finding container dbc27a8e987f01469e91fcc29387939652d2da779c28e3bed8b02821b48164d4: Status 404 returned error can't find the container with id dbc27a8e987f01469e91fcc29387939652d2da779c28e3bed8b02821b48164d4 Jul 26 15:59:40.507211 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:40.507180 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 15:59:40.508931 ip-10-0-139-15 kubenswrapper[2575]: W0726 15:59:40.508903 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff542bae_1bee_4eb1_bd6b_59b086f7648b.slice/crio-09b37a97a88b8cf64106ad30bc93e115a5586d18b47476811f99193a0fae253a WatchSource:0}: Error finding container 09b37a97a88b8cf64106ad30bc93e115a5586d18b47476811f99193a0fae253a: Status 404 returned error can't find the container with id 09b37a97a88b8cf64106ad30bc93e115a5586d18b47476811f99193a0fae253a Jul 26 15:59:41.215334 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.215289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" event={"ID":"ccbdb079-71e3-44b8-b33a-62d965f05dfe","Type":"ContainerStarted","Data":"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53"} Jul 26 15:59:41.215334 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.215342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" event={"ID":"ccbdb079-71e3-44b8-b33a-62d965f05dfe","Type":"ContainerStarted","Data":"dbc27a8e987f01469e91fcc29387939652d2da779c28e3bed8b02821b48164d4"} Jul 26 15:59:41.215863 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.215484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:41.216908 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.216884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" event={"ID":"ff542bae-1bee-4eb1-bd6b-59b086f7648b","Type":"ContainerStarted","Data":"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f"} Jul 26 15:59:41.217000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.216929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" event={"ID":"ff542bae-1bee-4eb1-bd6b-59b086f7648b","Type":"ContainerStarted","Data":"09b37a97a88b8cf64106ad30bc93e115a5586d18b47476811f99193a0fae253a"} Jul 26 15:59:41.217000 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.216960 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 15:59:41.234723 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.234673 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" podStartSLOduration=2.234659176 podStartE2EDuration="2.234659176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:41.233122751 +0000 UTC m=+3255.130797747" watchObservedRunningTime="2026-07-26 15:59:41.234659176 +0000 UTC m=+3255.132334212" Jul 26 15:59:41.253835 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:41.253789 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" podStartSLOduration=2.2537734 podStartE2EDuration="2.2537734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:59:41.25158497 +0000 UTC m=+3255.149259967" watchObservedRunningTime="2026-07-26 15:59:41.2537734 +0000 UTC m=+3255.151448397" Jul 26 15:59:44.733019 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:44.732970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 15:59:52.226598 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:52.226560 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 15:59:52.227163 ip-10-0-139-15 kubenswrapper[2575]: I0726 15:59:52.226654 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 16:00:04.922990 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:04.922952 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 16:00:04.923462 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:04.923192 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" podUID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" containerName="payload-processing" containerID="cri-o://4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f" gracePeriod=30 Jul 26 16:00:04.943495 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:04.943464 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 16:00:04.943678 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:04.943656 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" podUID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" containerName="payload-pre-processing" containerID="cri-o://5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53" gracePeriod=30 Jul 26 16:00:05.220985 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.220958 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 16:00:05.256462 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.256430 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 16:00:05.347724 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.347685 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume\") pod \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " Jul 26 16:00:05.347896 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.347829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gff74\" (UniqueName: \"kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74\") pod \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\" (UID: \"ff542bae-1bee-4eb1-bd6b-59b086f7648b\") " Jul 26 16:00:05.347896 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.347864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5khcw\" (UniqueName: \"kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw\") pod \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " Jul 26 16:00:05.347896 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.347882 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume\") pod \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\" (UID: \"ccbdb079-71e3-44b8-b33a-62d965f05dfe\") " Jul 26 16:00:05.348126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.348099 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff542bae-1bee-4eb1-bd6b-59b086f7648b" (UID: "ff542bae-1bee-4eb1-bd6b-59b086f7648b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:00:05.348367 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.348337 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ccbdb079-71e3-44b8-b33a-62d965f05dfe" (UID: "ccbdb079-71e3-44b8-b33a-62d965f05dfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:00:05.350172 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.350154 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw" (OuterVolumeSpecName: "kube-api-access-5khcw") pod "ccbdb079-71e3-44b8-b33a-62d965f05dfe" (UID: "ccbdb079-71e3-44b8-b33a-62d965f05dfe"). InnerVolumeSpecName "kube-api-access-5khcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:00:05.350241 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.350182 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74" (OuterVolumeSpecName: "kube-api-access-gff74") pod "ff542bae-1bee-4eb1-bd6b-59b086f7648b" (UID: "ff542bae-1bee-4eb1-bd6b-59b086f7648b"). InnerVolumeSpecName "kube-api-access-gff74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:00:05.371420 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.371389 2575 generic.go:356] "Generic (PLEG): container finished" podID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" containerID="5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53" exitCode=0 Jul 26 16:00:05.371620 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.371431 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" Jul 26 16:00:05.371620 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.371470 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" event={"ID":"ccbdb079-71e3-44b8-b33a-62d965f05dfe","Type":"ContainerDied","Data":"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53"} Jul 26 16:00:05.371620 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.371526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j" event={"ID":"ccbdb079-71e3-44b8-b33a-62d965f05dfe","Type":"ContainerDied","Data":"dbc27a8e987f01469e91fcc29387939652d2da779c28e3bed8b02821b48164d4"} Jul 26 16:00:05.371620 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.371552 2575 scope.go:117] "RemoveContainer" containerID="5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53" Jul 26 16:00:05.372900 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.372876 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" containerID="4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f" exitCode=0 Jul 26 16:00:05.373006 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.372902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" event={"ID":"ff542bae-1bee-4eb1-bd6b-59b086f7648b","Type":"ContainerDied","Data":"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f"} Jul 26 16:00:05.373006 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.372931 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" Jul 26 16:00:05.373006 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.372959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94" event={"ID":"ff542bae-1bee-4eb1-bd6b-59b086f7648b","Type":"ContainerDied","Data":"09b37a97a88b8cf64106ad30bc93e115a5586d18b47476811f99193a0fae253a"} Jul 26 16:00:05.383375 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.383350 2575 scope.go:117] "RemoveContainer" containerID="5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53" Jul 26 16:00:05.383821 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:00:05.383775 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53\": container with ID starting with 5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53 not found: ID does not exist" containerID="5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53" Jul 26 16:00:05.383927 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.383830 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53"} err="failed to get container status \"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53\": rpc error: code = NotFound desc = could not find container \"5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53\": container with ID starting with 5ee36b04729feefa5f3f876f73412545810383133a93b8456f566162a835ce53 not found: ID does not exist" Jul 26 16:00:05.383927 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.383855 2575 scope.go:117] "RemoveContainer" containerID="4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f" Jul 26 16:00:05.393744 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.393724 2575 scope.go:117] "RemoveContainer" containerID="4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f" Jul 26 16:00:05.393996 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:00:05.393970 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f\": container with ID starting with 4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f not found: ID does not exist" containerID="4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f" Jul 26 16:00:05.394085 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.394004 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f"} err="failed to get container status \"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f\": rpc error: code = NotFound desc = could not find container \"4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f\": container with ID starting with 4b77c83baea6a4ac4602e20a4c5afa744aa326f0a58489dd6416d30007087a0f not found: ID does not exist" Jul 26 16:00:05.398525 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.398503 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 16:00:05.402699 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.402674 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-35252a-568598db64-m4d4j"] Jul 26 16:00:05.412981 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.412955 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 16:00:05.415366 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.415347 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-35252a-56dc5d75f6-gdj94"] Jul 26 16:00:05.448631 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.448558 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gff74\" (UniqueName: \"kubernetes.io/projected/ff542bae-1bee-4eb1-bd6b-59b086f7648b-kube-api-access-gff74\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.448631 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.448584 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khcw\" (UniqueName: \"kubernetes.io/projected/ccbdb079-71e3-44b8-b33a-62d965f05dfe-kube-api-access-5khcw\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.448631 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.448594 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbdb079-71e3-44b8-b33a-62d965f05dfe-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:05.448631 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:05.448603 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff542bae-1bee-4eb1-bd6b-59b086f7648b-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:06.561519 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:06.561481 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" path="/var/lib/kubelet/pods/ccbdb079-71e3-44b8-b33a-62d965f05dfe/volumes" Jul 26 16:00:06.561911 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:06.561851 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" path="/var/lib/kubelet/pods/ff542bae-1bee-4eb1-bd6b-59b086f7648b/volumes" Jul 26 16:00:09.842614 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:09.842572 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:00:30.694304 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:30.694250 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 16:00:30.704490 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:30.704463 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 16:00:34.130600 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:34.130564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:00:35.162317 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.160686 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 16:00:35.162317 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.161034 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" podUID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" containerName="payload-processing" containerID="cri-o://098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2" gracePeriod=30 Jul 26 16:00:35.447573 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.447548 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 16:00:35.547483 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.547449 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pgzg\" (UniqueName: \"kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg\") pod \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " Jul 26 16:00:35.547483 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.547488 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume\") pod \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\" (UID: \"4df9add7-5ee4-42c9-8e1a-15374dfedb17\") " Jul 26 16:00:35.547855 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.547832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4df9add7-5ee4-42c9-8e1a-15374dfedb17" (UID: "4df9add7-5ee4-42c9-8e1a-15374dfedb17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:00:35.549743 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.549718 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg" (OuterVolumeSpecName: "kube-api-access-5pgzg") pod "4df9add7-5ee4-42c9-8e1a-15374dfedb17" (UID: "4df9add7-5ee4-42c9-8e1a-15374dfedb17"). InnerVolumeSpecName "kube-api-access-5pgzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:00:35.561209 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.561175 2575 generic.go:356] "Generic (PLEG): container finished" podID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" containerID="098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2" exitCode=0 Jul 26 16:00:35.561350 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.561215 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" Jul 26 16:00:35.561350 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.561253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" event={"ID":"4df9add7-5ee4-42c9-8e1a-15374dfedb17","Type":"ContainerDied","Data":"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2"} Jul 26 16:00:35.561350 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.561342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8" event={"ID":"4df9add7-5ee4-42c9-8e1a-15374dfedb17","Type":"ContainerDied","Data":"1c5a125515c3055cd144ddc36ad64b58b0567215acfe46ff643d2bce8488b4ac"} Jul 26 16:00:35.561466 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.561373 2575 scope.go:117] "RemoveContainer" containerID="098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2" Jul 26 16:00:35.571010 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.570988 2575 scope.go:117] "RemoveContainer" containerID="098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2" Jul 26 16:00:35.571267 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:00:35.571251 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2\": container with ID starting with 098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2 not found: ID does not exist" containerID="098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2" Jul 26 16:00:35.571329 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.571309 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2"} err="failed to get container status \"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2\": rpc error: code = NotFound desc = could not find container \"098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2\": container with ID starting with 098069b35f1557e02ce6182ccd5f291d690fdee3de10f4f93c118022503a75f2 not found: ID does not exist" Jul 26 16:00:35.582813 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.582782 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 16:00:35.588242 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.588214 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d57e1d-bff84c4b-nvvr8"] Jul 26 16:00:35.648269 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.648232 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgzg\" (UniqueName: \"kubernetes.io/projected/4df9add7-5ee4-42c9-8e1a-15374dfedb17-kube-api-access-5pgzg\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:35.648269 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:35.648265 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4df9add7-5ee4-42c9-8e1a-15374dfedb17-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:00:36.554822 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:36.554785 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" path="/var/lib/kubelet/pods/4df9add7-5ee4-42c9-8e1a-15374dfedb17/volumes" Jul 26 16:00:44.833685 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:00:44.833643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:01:33.560389 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:33.560350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 16:01:33.560835 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:33.560554 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" containerID="cri-o://80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35" gracePeriod=30 Jul 26 16:01:38.825342 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.825243 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 16:01:38.941233 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.941194 2575 generic.go:356] "Generic (PLEG): container finished" podID="87efd397-f344-4908-b3ad-47cd1c406759" containerID="80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35" exitCode=0 Jul 26 16:01:38.941467 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.941250 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" Jul 26 16:01:38.941467 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.941314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerDied","Data":"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35"} Jul 26 16:01:38.941467 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.941385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg" event={"ID":"87efd397-f344-4908-b3ad-47cd1c406759","Type":"ContainerDied","Data":"a375fb7d2ddf692ebdd8aa56ec3a0c485d565cf7324ca9fec19d3bece401a1e6"} Jul 26 16:01:38.941467 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.941417 2575 scope.go:117] "RemoveContainer" containerID="80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35" Jul 26 16:01:38.951232 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.951208 2575 scope.go:117] "RemoveContainer" containerID="f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3" Jul 26 16:01:38.960305 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.960262 2575 scope.go:117] "RemoveContainer" containerID="80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35" Jul 26 16:01:38.960607 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:01:38.960588 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35\": container with ID starting with 80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35 not found: ID does not exist" containerID="80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35" Jul 26 16:01:38.960666 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.960617 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35"} err="failed to get container status \"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35\": rpc error: code = NotFound desc = could not find container \"80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35\": container with ID starting with 80da69d84030c6b9f5b53b83d1266121249c700249a2094248ad0c240def3b35 not found: ID does not exist" Jul 26 16:01:38.960666 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.960636 2575 scope.go:117] "RemoveContainer" containerID="f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3" Jul 26 16:01:38.960946 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:01:38.960927 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3\": container with ID starting with f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3 not found: ID does not exist" containerID="f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3" Jul 26 16:01:38.961004 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.960952 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3"} err="failed to get container status \"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3\": rpc error: code = NotFound desc = could not find container \"f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3\": container with ID starting with f4ac1845febeaf5df385096df9efb05d0f971b5bd01242c8e9685f1d1413e5d3 not found: ID does not exist" Jul 26 16:01:38.969380 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969509 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969509 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969449 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969509 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969498 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969524 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969554 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbwln\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969588 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969634 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert\") pod \"87efd397-f344-4908-b3ad-47cd1c406759\" (UID: \"87efd397-f344-4908-b3ad-47cd1c406759\") " Jul 26 16:01:38.969948 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:38.969948 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969794 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:38.970073 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.969990 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:38.970073 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.970014 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:38.970073 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.970032 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:38.970073 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.970062 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:01:38.970608 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.970585 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data" (OuterVolumeSpecName: "istio-data") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:38.971933 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.971900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln" (OuterVolumeSpecName: "kube-api-access-mbwln") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "kube-api-access-mbwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:01:38.972312 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.972292 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token" (OuterVolumeSpecName: "istio-token") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:01:38.972312 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.972301 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:01:38.972421 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:38.972386 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87efd397-f344-4908-b3ad-47cd1c406759" (UID: "87efd397-f344-4908-b3ad-47cd1c406759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:01:39.070849 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070808 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87efd397-f344-4908-b3ad-47cd1c406759-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.070849 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070842 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.070849 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070850 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwln\" (UniqueName: \"kubernetes.io/projected/87efd397-f344-4908-b3ad-47cd1c406759-kube-api-access-mbwln\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.070849 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070862 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.071128 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070870 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87efd397-f344-4908-b3ad-47cd1c406759-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.071128 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.070877 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87efd397-f344-4908-b3ad-47cd1c406759-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:01:39.263845 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.263814 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 16:01:39.268322 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:39.268297 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d57e1d-openshift-default-c6cdbbd8b-2mzkg"] Jul 26 16:01:40.554874 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:40.554834 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87efd397-f344-4908-b3ad-47cd1c406759" path="/var/lib/kubelet/pods/87efd397-f344-4908-b3ad-47cd1c406759/volumes" Jul 26 16:01:47.668430 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:47.668384 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:01:55.379247 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:55.379206 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 16:01:55.379753 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:01:55.379480 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" containerID="cri-o://005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469" gracePeriod=30 Jul 26 16:02:00.643486 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.643414 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 16:02:00.673632 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.673818 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673643 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.673818 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673684 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.673818 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673705 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.673818 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.673818 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673815 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nz9g\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.674093 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.674093 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.674093 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.673922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert\") pod \"61bc15aa-02e5-4838-bc86-522eb72c44c4\" (UID: \"61bc15aa-02e5-4838-bc86-522eb72c44c4\") " Jul 26 16:02:00.674535 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.674510 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:00.674898 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.674813 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:00.674898 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.674523 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:00.674898 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.674668 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:00.675149 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.674985 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:00.676635 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.676607 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:00.676829 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.676803 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:02:00.676896 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.676822 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g" (OuterVolumeSpecName: "kube-api-access-6nz9g") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "kube-api-access-6nz9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:00.676936 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.676925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61bc15aa-02e5-4838-bc86-522eb72c44c4" (UID: "61bc15aa-02e5-4838-bc86-522eb72c44c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:00.775542 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775501 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775542 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775537 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nz9g\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-kube-api-access-6nz9g\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775542 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775547 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775557 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775565 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bc15aa-02e5-4838-bc86-522eb72c44c4-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775573 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775580 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bc15aa-02e5-4838-bc86-522eb72c44c4-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775588 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:00.775788 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:00.775596 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bc15aa-02e5-4838-bc86-522eb72c44c4-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:01.077632 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.077594 2575 generic.go:356] "Generic (PLEG): container finished" podID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerID="005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469" exitCode=0 Jul 26 16:02:01.077841 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.077663 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" Jul 26 16:02:01.077841 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.077675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerDied","Data":"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469"} Jul 26 16:02:01.077841 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.077758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz" event={"ID":"61bc15aa-02e5-4838-bc86-522eb72c44c4","Type":"ContainerDied","Data":"7e4bd60e25ff37618793cfbe6b70cd255e5de02834f5edb0abcddae0c0436a8b"} Jul 26 16:02:01.077841 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.077783 2575 scope.go:117] "RemoveContainer" containerID="005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469" Jul 26 16:02:01.088073 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.088053 2575 scope.go:117] "RemoveContainer" containerID="ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741" Jul 26 16:02:01.098117 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.098095 2575 scope.go:117] "RemoveContainer" containerID="005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469" Jul 26 16:02:01.098468 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:01.098444 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469\": container with ID starting with 005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469 not found: ID does not exist" containerID="005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469" Jul 26 16:02:01.098964 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.098507 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469"} err="failed to get container status \"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469\": rpc error: code = NotFound desc = could not find container \"005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469\": container with ID starting with 005ec9fcc09bb56f66f33fc69415457dc418136e625fde5c215f70460a125469 not found: ID does not exist" Jul 26 16:02:01.098964 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.098565 2575 scope.go:117] "RemoveContainer" containerID="ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741" Jul 26 16:02:01.099163 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:01.099032 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741\": container with ID starting with ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741 not found: ID does not exist" containerID="ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741" Jul 26 16:02:01.099163 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.099065 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741"} err="failed to get container status \"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741\": rpc error: code = NotFound desc = could not find container \"ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741\": container with ID starting with ff5b835e05b763ff5b4ce72ecd7e0c0d6f07213ed3707f0316b66249417d5741 not found: ID does not exist" Jul 26 16:02:01.100676 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.100651 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 16:02:01.103051 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:01.103031 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cb6b99-openshift-default-6f5bbbcdfd-vxnsz"] Jul 26 16:02:02.555022 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:02.554989 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" path="/var/lib/kubelet/pods/61bc15aa-02e5-4838-bc86-522eb72c44c4/volumes" Jul 26 16:02:06.359575 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.359542 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 16:02:06.359966 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.359757 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" podUID="5834b42a-39a0-446d-9a02-b8d18958a4c4" containerName="payload-pre-processing" containerID="cri-o://30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd" gracePeriod=30 Jul 26 16:02:06.627040 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.627011 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 16:02:06.730955 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.730916 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh\") pod \"5834b42a-39a0-446d-9a02-b8d18958a4c4\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " Jul 26 16:02:06.731130 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.730976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume\") pod \"5834b42a-39a0-446d-9a02-b8d18958a4c4\" (UID: \"5834b42a-39a0-446d-9a02-b8d18958a4c4\") " Jul 26 16:02:06.731443 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.731416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5834b42a-39a0-446d-9a02-b8d18958a4c4" (UID: "5834b42a-39a0-446d-9a02-b8d18958a4c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:06.733284 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.733256 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh" (OuterVolumeSpecName: "kube-api-access-zq9jh") pod "5834b42a-39a0-446d-9a02-b8d18958a4c4" (UID: "5834b42a-39a0-446d-9a02-b8d18958a4c4"). InnerVolumeSpecName "kube-api-access-zq9jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:06.832412 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.832372 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq9jh\" (UniqueName: \"kubernetes.io/projected/5834b42a-39a0-446d-9a02-b8d18958a4c4-kube-api-access-zq9jh\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:06.832412 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:06.832404 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5834b42a-39a0-446d-9a02-b8d18958a4c4-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:07.115484 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.115444 2575 generic.go:356] "Generic (PLEG): container finished" podID="5834b42a-39a0-446d-9a02-b8d18958a4c4" containerID="30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd" exitCode=0 Jul 26 16:02:07.115655 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.115510 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" Jul 26 16:02:07.115655 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.115528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" event={"ID":"5834b42a-39a0-446d-9a02-b8d18958a4c4","Type":"ContainerDied","Data":"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd"} Jul 26 16:02:07.115655 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.115613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm" event={"ID":"5834b42a-39a0-446d-9a02-b8d18958a4c4","Type":"ContainerDied","Data":"7dc6fd5198e4787bff83e4a3a17de16c984845892f7a756fbddbeaa0453a7d99"} Jul 26 16:02:07.115655 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.115636 2575 scope.go:117] "RemoveContainer" containerID="30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd" Jul 26 16:02:07.126012 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.125993 2575 scope.go:117] "RemoveContainer" containerID="30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd" Jul 26 16:02:07.126262 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:07.126245 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd\": container with ID starting with 30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd not found: ID does not exist" containerID="30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd" Jul 26 16:02:07.126429 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.126270 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd"} err="failed to get container status \"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd\": rpc error: code = NotFound desc = could not find container \"30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd\": container with ID starting with 30c7c2cbfc4e919e192f3b407bf7fc7b9773898e70a83418ea3bfa99979098fd not found: ID does not exist" Jul 26 16:02:07.138234 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.138207 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 16:02:07.141367 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:07.141340 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a20d61-59c9dd55fb-h8wcm"] Jul 26 16:02:08.555139 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:08.555103 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5834b42a-39a0-446d-9a02-b8d18958a4c4" path="/var/lib/kubelet/pods/5834b42a-39a0-446d-9a02-b8d18958a4c4/volumes" Jul 26 16:02:09.727940 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:09.727902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:02:17.396952 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:17.396866 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 16:02:17.397428 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:17.397112 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" containerID="cri-o://1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa" gracePeriod=30 Jul 26 16:02:21.108402 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:21.108360 2575 patch_prober.go:28] interesting pod/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:02:21.108784 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:21.108427 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:02:22.670913 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.670828 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 16:02:22.787574 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787541 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787574 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787578 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787603 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787637 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787711 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787732 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5smd9\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787754 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.787815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787816 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs\") pod \"56664af8-cabb-42c7-8f19-c358bae60ddd\" (UID: \"56664af8-cabb-42c7-8f19-c358bae60ddd\") " Jul 26 16:02:22.788186 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.787920 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:22.788186 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.788120 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:22.788186 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.788129 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:22.788186 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.788179 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.788436 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.788298 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:22.788648 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.788499 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data" (OuterVolumeSpecName: "istio-data") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:22.790115 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.790080 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:22.790248 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.790136 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:02:22.790248 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.790152 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token" (OuterVolumeSpecName: "istio-token") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:22.790430 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.790317 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9" (OuterVolumeSpecName: "kube-api-access-5smd9") pod "56664af8-cabb-42c7-8f19-c358bae60ddd" (UID: "56664af8-cabb-42c7-8f19-c358bae60ddd"). InnerVolumeSpecName "kube-api-access-5smd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:22.888865 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888830 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.888865 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888861 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.888865 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888871 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56664af8-cabb-42c7-8f19-c358bae60ddd-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.889104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888879 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smd9\" (UniqueName: \"kubernetes.io/projected/56664af8-cabb-42c7-8f19-c358bae60ddd-kube-api-access-5smd9\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.889104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888888 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.889104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888896 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.889104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888904 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:22.889104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:22.888911 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56664af8-cabb-42c7-8f19-c358bae60ddd-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:23.213583 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.213549 2575 generic.go:356] "Generic (PLEG): container finished" podID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerID="1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa" exitCode=0 Jul 26 16:02:23.213786 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.213592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerDied","Data":"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa"} Jul 26 16:02:23.213786 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.213604 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" Jul 26 16:02:23.213786 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.213642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn" event={"ID":"56664af8-cabb-42c7-8f19-c358bae60ddd","Type":"ContainerDied","Data":"2f19b7f48e410bc076dd3c96719510a38e2cbe6f6e95e49ed6e6bbe108efaf01"} Jul 26 16:02:23.213786 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.213666 2575 scope.go:117] "RemoveContainer" containerID="1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa" Jul 26 16:02:23.223757 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.223737 2575 scope.go:117] "RemoveContainer" containerID="e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0" Jul 26 16:02:23.232760 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.232735 2575 scope.go:117] "RemoveContainer" containerID="1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa" Jul 26 16:02:23.233045 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:23.233025 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa\": container with ID starting with 1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa not found: ID does not exist" containerID="1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa" Jul 26 16:02:23.233084 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.233054 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa"} err="failed to get container status \"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa\": rpc error: code = NotFound desc = could not find container \"1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa\": container with ID starting with 1787ae1202c40aef40c1bdfa173dfa941b92624ec41bad3ceca5ef2cc391ceaa not found: ID does not exist" Jul 26 16:02:23.233084 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.233071 2575 scope.go:117] "RemoveContainer" containerID="e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0" Jul 26 16:02:23.233414 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:23.233367 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0\": container with ID starting with e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0 not found: ID does not exist" containerID="e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0" Jul 26 16:02:23.233414 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.233396 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0"} err="failed to get container status \"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0\": rpc error: code = NotFound desc = could not find container \"e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0\": container with ID starting with e5aaccd94ddb94acbeed38beca9a540c024ad1680a2d942d69642aef3b8220e0 not found: ID does not exist" Jul 26 16:02:23.236702 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.236673 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 16:02:23.243952 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:23.243928 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a20d61-openshift-default-56f4594668-zsngn"] Jul 26 16:02:24.554936 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:24.554901 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" path="/var/lib/kubelet/pods/56664af8-cabb-42c7-8f19-c358bae60ddd/volumes" Jul 26 16:02:28.387709 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.387672 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 16:02:28.388288 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.387915 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" podUID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" containerName="payload-pre-processing" containerID="cri-o://ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756" gracePeriod=30 Jul 26 16:02:28.662755 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.662716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 16:02:28.850199 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.850153 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb4dz\" (UniqueName: \"kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz\") pod \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " Jul 26 16:02:28.850410 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.850219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume\") pod \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\" (UID: \"6f4a569b-cb59-4dd1-bed8-898d03b6c891\") " Jul 26 16:02:28.850669 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.850640 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f4a569b-cb59-4dd1-bed8-898d03b6c891" (UID: "6f4a569b-cb59-4dd1-bed8-898d03b6c891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:28.852419 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.852394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz" (OuterVolumeSpecName: "kube-api-access-hb4dz") pod "6f4a569b-cb59-4dd1-bed8-898d03b6c891" (UID: "6f4a569b-cb59-4dd1-bed8-898d03b6c891"). InnerVolumeSpecName "kube-api-access-hb4dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:28.951061 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.951023 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4dz\" (UniqueName: \"kubernetes.io/projected/6f4a569b-cb59-4dd1-bed8-898d03b6c891-kube-api-access-hb4dz\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:28.951061 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:28.951054 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f4a569b-cb59-4dd1-bed8-898d03b6c891-plugins-config-volume\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:29.257149 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.257063 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" containerID="ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756" exitCode=0 Jul 26 16:02:29.257149 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.257107 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" Jul 26 16:02:29.257345 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.257152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" event={"ID":"6f4a569b-cb59-4dd1-bed8-898d03b6c891","Type":"ContainerDied","Data":"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756"} Jul 26 16:02:29.257345 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.257206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f" event={"ID":"6f4a569b-cb59-4dd1-bed8-898d03b6c891","Type":"ContainerDied","Data":"edf8c738f4e903ed4ad5f9f39675409c222297e1327b9af67f0da206528ffd1c"} Jul 26 16:02:29.257345 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.257228 2575 scope.go:117] "RemoveContainer" containerID="ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756" Jul 26 16:02:29.266893 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.266873 2575 scope.go:117] "RemoveContainer" containerID="ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756" Jul 26 16:02:29.267172 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:29.267156 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756\": container with ID starting with ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756 not found: ID does not exist" containerID="ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756" Jul 26 16:02:29.267216 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.267181 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756"} err="failed to get container status \"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756\": rpc error: code = NotFound desc = could not find container \"ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756\": container with ID starting with ba89f10344bb68df7ae26bd02c2c5aae338b2cf262adad5e110992f6adc60756 not found: ID does not exist" Jul 26 16:02:29.278515 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.278477 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 16:02:29.283646 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:29.283619 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-60c814-78c84b8cb4-xkq9f"] Jul 26 16:02:30.554204 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:30.554165 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" path="/var/lib/kubelet/pods/6f4a569b-cb59-4dd1-bed8-898d03b6c891/volumes" Jul 26 16:02:31.434565 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:31.434531 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-sv7ch"] Jul 26 16:02:39.361999 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:39.361963 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 16:02:39.362577 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:39.362187 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" containerID="cri-o://494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc" gracePeriod=30 Jul 26 16:02:43.354527 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:43.354487 2575 patch_prober.go:28] interesting pod/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:02:43.354963 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:43.354558 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:02:44.636360 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.636256 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 16:02:44.689677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689647 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.689844 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689692 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.689844 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.689844 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689784 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbssw\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.689844 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689810 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.690052 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689954 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.690052 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.689969 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:44.690052 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.690052 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690047 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.690252 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket\") pod \"bb49da13-5eb5-4a86-8499-2181761aec33\" (UID: \"bb49da13-5eb5-4a86-8499-2181761aec33\") " Jul 26 16:02:44.690252 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690238 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:02:44.690562 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690493 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:44.690562 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690526 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-credential-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.690562 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690546 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb49da13-5eb5-4a86-8499-2181761aec33-istiod-ca-cert\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.690732 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.690584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:44.691050 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.691012 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:44.692150 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.692122 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:02:44.692327 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.692204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:44.692587 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.692569 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:02:44.692587 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.692578 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw" (OuterVolumeSpecName: "kube-api-access-rbssw") pod "bb49da13-5eb5-4a86-8499-2181761aec33" (UID: "bb49da13-5eb5-4a86-8499-2181761aec33"). InnerVolumeSpecName "kube-api-access-rbssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:02:44.791742 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791698 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb49da13-5eb5-4a86-8499-2181761aec33-istio-podinfo\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.791742 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791734 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbssw\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-kube-api-access-rbssw\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.791742 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791746 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-data\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.791742 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791754 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb49da13-5eb5-4a86-8499-2181761aec33-istio-token\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.792025 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791763 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-certs\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.792025 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791771 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-workload-socket\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:44.792025 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:44.791779 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb49da13-5eb5-4a86-8499-2181761aec33-istio-envoy\") on node \"ip-10-0-139-15.ec2.internal\" DevicePath \"\"" Jul 26 16:02:45.340115 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.340085 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-95c7j_c3c34933-b27a-4d7a-8c3c-d2c45d1476e7/manager/0.log" Jul 26 16:02:45.359194 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.359159 2575 generic.go:356] "Generic (PLEG): container finished" podID="bb49da13-5eb5-4a86-8499-2181761aec33" containerID="494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc" exitCode=0 Jul 26 16:02:45.359417 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.359215 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" Jul 26 16:02:45.359417 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.359251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerDied","Data":"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc"} Jul 26 16:02:45.359417 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.359364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7" event={"ID":"bb49da13-5eb5-4a86-8499-2181761aec33","Type":"ContainerDied","Data":"f75f479efa7349b09a9b4fe0651c4598c18b527b0992531915516f0fc46c5340"} Jul 26 16:02:45.359417 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.359394 2575 scope.go:117] "RemoveContainer" containerID="494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc" Jul 26 16:02:45.369255 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.369234 2575 scope.go:117] "RemoveContainer" containerID="c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6" Jul 26 16:02:45.378310 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.378262 2575 scope.go:117] "RemoveContainer" containerID="494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc" Jul 26 16:02:45.378616 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:45.378597 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc\": container with ID starting with 494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc not found: ID does not exist" containerID="494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc" Jul 26 16:02:45.378679 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.378625 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc"} err="failed to get container status \"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc\": rpc error: code = NotFound desc = could not find container \"494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc\": container with ID starting with 494c75769f2e60d2e22de9a5733ec2f1556abab9d8a7cdfb5d7596ca1a4934bc not found: ID does not exist" Jul 26 16:02:45.378679 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.378646 2575 scope.go:117] "RemoveContainer" containerID="c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6" Jul 26 16:02:45.378868 ip-10-0-139-15 kubenswrapper[2575]: E0726 16:02:45.378852 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6\": container with ID starting with c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6 not found: ID does not exist" containerID="c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6" Jul 26 16:02:45.378916 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.378875 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6"} err="failed to get container status \"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6\": rpc error: code = NotFound desc = could not find container \"c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6\": container with ID starting with c10d62c6445e45384ab72b9ec12cde6113cd076ee399b92e10c980020cbf64c6 not found: ID does not exist" Jul 26 16:02:45.388993 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.388963 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 16:02:45.391254 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.391218 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-60c814-openshift-default-7c545b87d4-jfcw7"] Jul 26 16:02:45.455831 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.455798 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lk99z_f9ead486-31bc-4c2a-a673-e05b7ae1e5ff/manager/0.log" Jul 26 16:02:45.823003 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.822968 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdtfj_9376abe2-626d-44f6-8998-5e30259db5fc/manager/0.log" Jul 26 16:02:45.939203 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.939160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-578rs_fa8cacf2-24fd-41eb-90a9-01590afc146c/manager/0.log" Jul 26 16:02:45.944780 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:45.944759 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-578rs_fa8cacf2-24fd-41eb-90a9-01590afc146c/copy-manifests/0.log" Jul 26 16:02:46.556215 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:46.556175 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" path="/var/lib/kubelet/pods/bb49da13-5eb5-4a86-8499-2181761aec33/volumes" Jul 26 16:02:46.635708 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:46.635675 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751330-ltcts_c3e499fa-3191-4755-be03-4a125134604c/cleanup/0.log" Jul 26 16:02:46.959754 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:46.959728 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jzg62_dbc3428a-481c-4dfc-883a-e2175455b907/postgres/0.log" Jul 26 16:02:47.728107 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.728076 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/util/0.log" Jul 26 16:02:47.733922 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.733903 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/pull/0.log" Jul 26 16:02:47.739212 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.739173 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/extract/0.log" Jul 26 16:02:47.853233 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.853208 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/util/0.log" Jul 26 16:02:47.859564 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.859545 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/pull/0.log" Jul 26 16:02:47.866469 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.866447 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/extract/0.log" Jul 26 16:02:47.976455 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.976428 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/extract/0.log" Jul 26 16:02:47.982498 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.982424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/util/0.log" Jul 26 16:02:47.988248 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:47.988229 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/pull/0.log" Jul 26 16:02:48.098303 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.098251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/util/0.log" Jul 26 16:02:48.104656 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.104630 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/pull/0.log" Jul 26 16:02:48.110535 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.110514 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/extract/0.log" Jul 26 16:02:48.342081 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.341998 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wh8hh_3fb0e491-250d-4840-8da6-84203acbe577/manager/0.log" Jul 26 16:02:48.452314 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.452262 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qxfzm_1c65057d-2fc3-464c-9afb-e6e51d9b83d8/manager/0.log" Jul 26 16:02:48.672731 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.672647 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-45ccx_96472d48-c6f8-4c04-9489-aa7ff496c1e7/registry-server/0.log" Jul 26 16:02:48.800237 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.800190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k2wjb_f1a348c3-73b2-48af-83e6-af8db0a6ee62/manager/0.log" Jul 26 16:02:48.907610 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:48.907574 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sv7ch_c17242eb-5d81-4075-8f5c-b15b59534c81/limitador/0.log" Jul 26 16:02:49.593165 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:49.593124 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-tzznd_3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a/kube-auth-proxy/0.log" Jul 26 16:02:49.828539 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:49.828496 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/1.log" Jul 26 16:02:49.948566 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:49.948537 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-68zjv_2bfc48ed-b26c-41be-898a-478c34b4651f/payload-pre-processing/0.log" Jul 26 16:02:50.615565 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:50.615531 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k4nrr_74c621f9-1ad4-4260-aff0-e57ecc72b0ea/main/0.log" Jul 26 16:02:56.602263 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602221 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ldhb/must-gather-shl7g"] Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602590 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602602 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602615 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602620 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602630 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" containerName="payload-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602636 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" containerName="payload-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602645 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602650 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602658 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" containerName="payload-pre-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602663 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" containerName="payload-pre-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602669 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5834b42a-39a0-446d-9a02-b8d18958a4c4" containerName="payload-pre-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602674 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5834b42a-39a0-446d-9a02-b8d18958a4c4" containerName="payload-pre-processing" Jul 26 16:02:56.602677 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602681 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602687 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602695 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602699 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602706 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" containerName="payload-pre-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602710 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" containerName="payload-pre-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602721 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" containerName="payload-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602727 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" containerName="payload-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602795 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5834b42a-39a0-446d-9a02-b8d18958a4c4" containerName="payload-pre-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602803 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602809 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602814 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff542bae-1bee-4eb1-bd6b-59b086f7648b" containerName="payload-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602820 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602825 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602830 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df9add7-5ee4-42c9-8e1a-15374dfedb17" containerName="payload-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602836 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602841 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbdb079-71e3-44b8-b33a-62d965f05dfe" containerName="payload-pre-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602846 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4a569b-cb59-4dd1-bed8-898d03b6c891" containerName="payload-pre-processing" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602894 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602901 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602916 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602921 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602934 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602939 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87efd397-f344-4908-b3ad-47cd1c406759" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602983 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb49da13-5eb5-4a86-8499-2181761aec33" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602988 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61bc15aa-02e5-4838-bc86-522eb72c44c4" containerName="istio-proxy" Jul 26 16:02:56.603126 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.602995 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56664af8-cabb-42c7-8f19-c358bae60ddd" containerName="istio-proxy" Jul 26 16:02:56.607535 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.607516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.610206 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.610185 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ldhb\"/\"default-dockercfg-5tfm4\"" Jul 26 16:02:56.610472 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.610453 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ldhb\"/\"kube-root-ca.crt\"" Jul 26 16:02:56.610715 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.610700 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ldhb\"/\"openshift-service-ca.crt\"" Jul 26 16:02:56.617217 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.617188 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/must-gather-shl7g"] Jul 26 16:02:56.702372 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.702338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdx7\" (UniqueName: \"kubernetes.io/projected/c29fdfb8-29a6-4f11-bb81-0d6752365b75-kube-api-access-sxdx7\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.702581 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.702552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c29fdfb8-29a6-4f11-bb81-0d6752365b75-must-gather-output\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.804029 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.803993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c29fdfb8-29a6-4f11-bb81-0d6752365b75-must-gather-output\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.804211 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.804039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdx7\" (UniqueName: \"kubernetes.io/projected/c29fdfb8-29a6-4f11-bb81-0d6752365b75-kube-api-access-sxdx7\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.804397 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.804372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c29fdfb8-29a6-4f11-bb81-0d6752365b75-must-gather-output\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.814059 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.814022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdx7\" (UniqueName: \"kubernetes.io/projected/c29fdfb8-29a6-4f11-bb81-0d6752365b75-kube-api-access-sxdx7\") pod \"must-gather-shl7g\" (UID: \"c29fdfb8-29a6-4f11-bb81-0d6752365b75\") " pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:56.919018 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:56.918920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/must-gather-shl7g" Jul 26 16:02:57.055876 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:57.055841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/must-gather-shl7g"] Jul 26 16:02:57.057185 ip-10-0-139-15 kubenswrapper[2575]: W0726 16:02:57.057157 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29fdfb8_29a6_4f11_bb81_0d6752365b75.slice/crio-df8e1dc55ac50d69273f12c936d5090f988d1ad5cc96b8c5a7b6af1da5a22097 WatchSource:0}: Error finding container df8e1dc55ac50d69273f12c936d5090f988d1ad5cc96b8c5a7b6af1da5a22097: Status 404 returned error can't find the container with id df8e1dc55ac50d69273f12c936d5090f988d1ad5cc96b8c5a7b6af1da5a22097 Jul 26 16:02:57.059129 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:57.059109 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:02:57.426959 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:57.426922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/must-gather-shl7g" event={"ID":"c29fdfb8-29a6-4f11-bb81-0d6752365b75","Type":"ContainerStarted","Data":"df8e1dc55ac50d69273f12c936d5090f988d1ad5cc96b8c5a7b6af1da5a22097"} Jul 26 16:02:58.437314 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:58.437241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/must-gather-shl7g" event={"ID":"c29fdfb8-29a6-4f11-bb81-0d6752365b75","Type":"ContainerStarted","Data":"9f45195b06384178700c156926b4b4c5297c8b2fbdb3a13a63f8983866ee8f51"} Jul 26 16:02:58.437760 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:58.437325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/must-gather-shl7g" event={"ID":"c29fdfb8-29a6-4f11-bb81-0d6752365b75","Type":"ContainerStarted","Data":"d00592ec0e7f4a9d1e49e31da9f812b3b7d63b757c6a308bb74270668dd7a4d9"} Jul 26 16:02:58.454443 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:58.454386 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4ldhb/must-gather-shl7g" podStartSLOduration=1.717445059 podStartE2EDuration="2.454367221s" totalImagesPullingTime="736.922162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:02:58.453375255 +0000 UTC m=+3452.351050255" watchObservedRunningTime="2026-07-26 16:02:58.454367221 +0000 UTC m=+3452.352042217" Jul 26 16:02:59.327473 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:59.327436 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k44rb_04d71036-db1b-40aa-9924-d767d55533b4/global-pull-secret-syncer/0.log" Jul 26 16:02:59.482295 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:59.482246 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xvmbh_fb496619-1f24-4fba-a0d2-cb1f942047df/konnectivity-agent/0.log" Jul 26 16:02:59.549182 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:02:59.549153 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-15.ec2.internal_a9f78f483deac2095511863fac8041de/haproxy/0.log" Jul 26 16:03:03.784752 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.784664 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/extract/0.log" Jul 26 16:03:03.808865 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.808827 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/util/0.log" Jul 26 16:03:03.841300 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.840155 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595snzs_5f79418b-8fd7-4226-b8a8-0c5472de6cdb/pull/0.log" Jul 26 16:03:03.883552 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.883516 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/extract/0.log" Jul 26 16:03:03.902878 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.902840 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/util/0.log" Jul 26 16:03:03.921424 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.921385 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b5pl9_0dccb90a-9e1e-4178-8781-5e0df1404e00/pull/0.log" Jul 26 16:03:03.973031 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.972964 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/extract/0.log" Jul 26 16:03:03.991748 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:03.991719 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/util/0.log" Jul 26 16:03:04.010550 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.010509 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7352gnr_f9e0a00d-05c2-4331-a69b-3b70ffe86777/pull/0.log" Jul 26 16:03:04.038098 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.037896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/extract/0.log" Jul 26 16:03:04.058882 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.058856 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/util/0.log" Jul 26 16:03:04.082807 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.082778 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14rlmh_615247b9-2b38-427b-bcbd-5c21ae1388ce/pull/0.log" Jul 26 16:03:04.718537 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.718506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wh8hh_3fb0e491-250d-4840-8da6-84203acbe577/manager/0.log" Jul 26 16:03:04.765171 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.765131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qxfzm_1c65057d-2fc3-464c-9afb-e6e51d9b83d8/manager/0.log" Jul 26 16:03:04.845347 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:04.845316 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-45ccx_96472d48-c6f8-4c04-9489-aa7ff496c1e7/registry-server/0.log" Jul 26 16:03:05.043358 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:05.043057 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k2wjb_f1a348c3-73b2-48af-83e6-af8db0a6ee62/manager/0.log" Jul 26 16:03:05.066693 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:05.066645 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-sv7ch_c17242eb-5d81-4075-8f5c-b15b59534c81/limitador/0.log" Jul 26 16:03:06.612714 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.612678 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/alertmanager/0.log" Jul 26 16:03:06.633637 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.633605 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/config-reloader/0.log" Jul 26 16:03:06.654965 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.654919 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/kube-rbac-proxy-web/0.log" Jul 26 16:03:06.673191 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.673160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/kube-rbac-proxy/0.log" Jul 26 16:03:06.703810 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.703779 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/kube-rbac-proxy-metric/0.log" Jul 26 16:03:06.725194 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.725163 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/prom-label-proxy/0.log" Jul 26 16:03:06.747872 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.747847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_926ce04b-980e-447c-b385-a84b190b9cab/init-config-reloader/0.log" Jul 26 16:03:06.831228 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.831191 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-shtnv_3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6/kube-state-metrics/0.log" Jul 26 16:03:06.851335 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.851303 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-shtnv_3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6/kube-rbac-proxy-main/0.log" Jul 26 16:03:06.887111 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.887029 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-shtnv_3cc2fd4a-7d0c-4c5c-a333-adac8a9c31e6/kube-rbac-proxy-self/0.log" Jul 26 16:03:06.987608 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:06.987580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b97j_59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e/node-exporter/0.log" Jul 26 16:03:07.019456 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:07.019426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b97j_59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e/kube-rbac-proxy/0.log" Jul 26 16:03:07.038029 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:07.037995 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2b97j_59ea7772-ae1f-4c28-8c64-20e3f9c5ba8e/init-textfile/0.log" Jul 26 16:03:09.724389 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.724353 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv"] Jul 26 16:03:09.729973 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.729937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.736930 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.736894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv"] Jul 26 16:03:09.746895 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.746603 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/1.log" Jul 26 16:03:09.755904 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.755825 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-n9tjl_715bac81-2670-4893-ba37-c3388bacf589/console-operator/2.log" Jul 26 16:03:09.851986 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.851936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-sys\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.852190 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.852020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf7f\" (UniqueName: \"kubernetes.io/projected/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-kube-api-access-mpf7f\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.852190 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.852080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-proc\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.852190 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.852109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-lib-modules\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.852190 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.852130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-podres\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953104 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf7f\" (UniqueName: \"kubernetes.io/projected/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-kube-api-access-mpf7f\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953339 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-proc\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953339 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-lib-modules\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953339 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-podres\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953547 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-proc\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953547 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-sys\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953547 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-lib-modules\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953547 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-podres\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.953785 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.953568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-sys\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:09.961261 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:09.961226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf7f\" (UniqueName: \"kubernetes.io/projected/3d0e9d17-519d-4966-af3c-83f5d0a3f7d7-kube-api-access-mpf7f\") pod \"perf-node-gather-daemonset-vfjlv\" (UID: \"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7\") " pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:10.043611 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.043508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:10.234131 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.233816 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv"] Jul 26 16:03:10.237572 ip-10-0-139-15 kubenswrapper[2575]: W0726 16:03:10.237536 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3d0e9d17_519d_4966_af3c_83f5d0a3f7d7.slice/crio-be8fe17601e097747b2e5fc29f223f2740134c4fd08974d93cd565536e88ad13 WatchSource:0}: Error finding container be8fe17601e097747b2e5fc29f223f2740134c4fd08974d93cd565536e88ad13: Status 404 returned error can't find the container with id be8fe17601e097747b2e5fc29f223f2740134c4fd08974d93cd565536e88ad13 Jul 26 16:03:10.356528 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.356490 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-vbg2b_fde2ca9c-4df1-4964-845f-86430deabdeb/download-server/0.log" Jul 26 16:03:10.546154 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.546115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" event={"ID":"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7","Type":"ContainerStarted","Data":"430bd9c7f7c2de25eb8f36e184b435ab4aa3e4f5f3750bf689bd2c07f5047484"} Jul 26 16:03:10.546367 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.546182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" event={"ID":"3d0e9d17-519d-4966-af3c-83f5d0a3f7d7","Type":"ContainerStarted","Data":"be8fe17601e097747b2e5fc29f223f2740134c4fd08974d93cd565536e88ad13"} Jul 26 16:03:10.546367 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.546245 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:10.563839 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:10.563731 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" podStartSLOduration=1.563715078 podStartE2EDuration="1.563715078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:03:10.561636274 +0000 UTC m=+3464.459311274" watchObservedRunningTime="2026-07-26 16:03:10.563715078 +0000 UTC m=+3464.461390074" Jul 26 16:03:11.997704 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:11.997655 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4s5l_96663026-7fcd-4267-a3f8-988a24bacacc/dns/0.log" Jul 26 16:03:12.014062 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:12.014034 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4s5l_96663026-7fcd-4267-a3f8-988a24bacacc/kube-rbac-proxy/0.log" Jul 26 16:03:12.034494 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:12.034463 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpdkp_fffe4393-063b-4937-97f4-35ace2c6f1d6/dns-node-resolver/0.log" Jul 26 16:03:12.586439 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:12.586368 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f56978d9b-hcfj8_7b012874-44e8-4a11-982b-02bed9f1bbcc/registry/0.log" Jul 26 16:03:12.645172 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:12.645143 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7sls_b39de062-6cbe-41fe-82e7-6133b46a5bca/node-ca/0.log" Jul 26 16:03:13.704562 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:13.704530 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-tzznd_3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a/kube-auth-proxy/0.log" Jul 26 16:03:13.751662 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:13.751625 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/0.log" Jul 26 16:03:13.787123 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:13.787090 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/1.log" Jul 26 16:03:13.815633 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:13.815596 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-68zjv_2bfc48ed-b26c-41be-898a-478c34b4651f/payload-pre-processing/0.log" Jul 26 16:03:14.678406 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:14.678369 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j967z_eab08770-3c75-4fad-bdeb-c5615fca0def/serve-healthcheck-canary/0.log" Jul 26 16:03:15.278692 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:15.278661 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jwj6g_62666105-e03c-446e-87dc-06c069722fa9/kube-rbac-proxy/0.log" Jul 26 16:03:15.295429 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:15.295391 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jwj6g_62666105-e03c-446e-87dc-06c069722fa9/exporter/0.log" Jul 26 16:03:15.314452 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:15.314419 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jwj6g_62666105-e03c-446e-87dc-06c069722fa9/extractor/0.log" Jul 26 16:03:16.564344 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:16.564314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ldhb/perf-node-gather-daemonset-vfjlv" Jul 26 16:03:17.450604 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:17.450569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-95c7j_c3c34933-b27a-4d7a-8c3c-d2c45d1476e7/manager/0.log" Jul 26 16:03:17.747904 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:17.747823 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lk99z_f9ead486-31bc-4c2a-a673-e05b7ae1e5ff/manager/0.log" Jul 26 16:03:18.069714 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:18.069559 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fdtfj_9376abe2-626d-44f6-8998-5e30259db5fc/manager/0.log" Jul 26 16:03:18.181996 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:18.181938 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-578rs_fa8cacf2-24fd-41eb-90a9-01590afc146c/manager/0.log" Jul 26 16:03:18.196356 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:18.196325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c55b8954b-578rs_fa8cacf2-24fd-41eb-90a9-01590afc146c/copy-manifests/0.log" Jul 26 16:03:19.755811 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:19.755770 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-sq5x9_8d071fcf-7aa9-429b-b1b7-3a51ee3ec747/openshift-lws-operator/0.log" Jul 26 16:03:20.761454 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:20.761420 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-tzznd_3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a/kube-auth-proxy/0.log" Jul 26 16:03:20.793807 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:20.793770 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/0.log" Jul 26 16:03:20.830321 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:20.830269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/1.log" Jul 26 16:03:20.860855 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:20.860816 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-68zjv_2bfc48ed-b26c-41be-898a-478c34b4651f/payload-pre-processing/0.log" Jul 26 16:03:22.487192 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:22.487152 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69f589b649-tzznd_3326be6d-eb7f-4f1c-a65b-00d82fc4ef7a/kube-auth-proxy/0.log" Jul 26 16:03:22.528535 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:22.528493 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/0.log" Jul 26 16:03:22.570261 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:22.570176 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-pfbpk_e8a474de-eaa9-4c6d-9a20-e8a442c86682/istio-proxy/1.log" Jul 26 16:03:22.596808 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:22.596773 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-68zjv_2bfc48ed-b26c-41be-898a-478c34b4651f/payload-pre-processing/0.log" Jul 26 16:03:26.163003 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.162969 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/kube-multus-additional-cni-plugins/0.log" Jul 26 16:03:26.180815 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.180780 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/egress-router-binary-copy/0.log" Jul 26 16:03:26.197592 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.197563 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/cni-plugins/0.log" Jul 26 16:03:26.214046 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.214004 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/bond-cni-plugin/0.log" Jul 26 16:03:26.232377 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.232345 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/routeoverride-cni/0.log" Jul 26 16:03:26.248636 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.248606 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/whereabouts-cni-bincopy/0.log" Jul 26 16:03:26.264943 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.264913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54ff4_5dd4f240-48ff-4c88-b6c4-d219767ed76c/whereabouts-cni/0.log" Jul 26 16:03:26.428819 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.428788 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fls4b_f137a282-0be2-47f7-aa1c-b2c5edae3249/kube-multus/0.log" Jul 26 16:03:26.521356 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.521327 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5ts4r_89b1a676-2fb9-4294-8a5a-6fd4e0c83da2/network-metrics-daemon/0.log" Jul 26 16:03:26.535582 ip-10-0-139-15 kubenswrapper[2575]: I0726 16:03:26.535554 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5ts4r_89b1a676-2fb9-4294-8a5a-6fd4e0c83da2/kube-rbac-proxy/0.log"