Jul 26 17:49:29.479083 ip-10-0-138-219 systemd[1]: Starting Kubernetes Kubelet... Jul 26 17:49:29.901317 ip-10-0-138-219 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 17:49:29.901317 ip-10-0-138-219 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 17:49:29.901317 ip-10-0-138-219 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 17:49:29.901317 ip-10-0-138-219 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 17:49:29.901317 ip-10-0-138-219 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 17:49:29.903412 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.903328 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 17:49:29.908606 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908588 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.908606 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908607 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908612 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908616 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908619 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908622 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908626 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908629 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908632 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908640 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908643 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908646 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908649 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908651 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908654 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908656 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908659 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908661 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908664 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908666 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908669 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.908680 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908671 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908674 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908677 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908679 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908682 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908686 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908689 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908692 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908695 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908698 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908701 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908703 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908706 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908709 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908711 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908715 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908720 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908724 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908726 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908729 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.909174 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908731 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908734 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908736 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908739 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908741 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908744 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908746 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908749 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908751 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908754 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908756 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908759 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908761 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908764 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908768 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908771 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908773 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908776 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908778 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.909673 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908781 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908784 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908798 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908801 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908803 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908806 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908809 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908812 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908814 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908817 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908819 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908822 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908825 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908828 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908830 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908833 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908836 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908838 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908841 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.910199 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908844 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908849 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908852 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908855 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908858 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908861 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908864 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908868 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908871 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908874 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908877 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908879 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908882 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908884 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908887 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908889 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908892 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908895 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908898 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.910649 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.908900 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909623 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909630 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909633 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909636 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909639 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909647 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909651 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909653 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909656 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909659 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909670 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909673 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909676 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909679 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909684 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909688 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909692 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909695 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.911233 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909698 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909700 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909705 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909709 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909713 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909717 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909720 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909722 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909725 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909728 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909731 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909733 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909736 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909738 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909741 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909743 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909746 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909748 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909751 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.911719 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909754 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909763 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909765 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909768 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909770 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909772 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909775 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909778 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909780 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909783 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909785 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909805 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909808 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909811 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909814 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909818 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909821 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909824 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909826 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909829 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.912209 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909832 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909834 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909837 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909839 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909842 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909844 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909847 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909849 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909852 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909854 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909857 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909860 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909862 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909864 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909868 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909870 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909873 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909875 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909878 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.912690 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909883 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909886 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909889 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909891 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909894 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909897 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909901 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909903 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909907 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909910 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909912 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909915 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909918 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909920 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909923 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909926 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909928 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909931 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909933 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909935 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.913167 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909938 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909940 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.909943 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910366 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910371 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910374 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910378 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910381 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910384 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910387 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910389 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910392 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910395 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910401 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910404 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910406 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910409 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910412 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910414 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910418 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.913659 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910423 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910427 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910430 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910433 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910435 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910438 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910440 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910443 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910472 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910491 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910497 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910503 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910506 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910509 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910513 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910516 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910519 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910522 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910525 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.914173 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910528 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910531 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910534 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910537 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910541 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910543 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910546 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910549 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910551 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910554 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910556 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910559 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910562 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910565 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910567 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910570 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910572 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910575 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910577 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910580 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.914635 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910582 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910585 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910589 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910592 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910595 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910597 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910600 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910603 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910605 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910608 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910611 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910613 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910615 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910618 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910621 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910623 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910626 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910628 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910631 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910634 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.915169 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910636 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910639 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910641 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910644 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910647 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910649 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910652 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910655 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910658 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910661 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910664 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910666 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910671 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910673 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910679 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910683 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910686 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910689 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910692 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.915648 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910694 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910698 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910704 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.910707 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910802 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910811 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910819 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910825 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910829 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910832 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910837 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910842 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910845 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910848 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910852 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910855 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910858 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910862 2575 flags.go:64] FLAG: --cgroup-root="" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910865 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910868 2575 flags.go:64] FLAG: --client-ca-file="" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910871 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910874 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910879 2575 flags.go:64] FLAG: --cluster-domain="" Jul 26 17:49:29.916130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910881 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910884 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910888 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910891 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910897 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910900 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910904 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910907 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910910 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910913 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910917 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910920 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910923 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910927 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910930 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910933 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910936 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910939 2575 flags.go:64] FLAG: --enable-server="true" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910942 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910947 2575 flags.go:64] FLAG: --event-burst="100" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910950 2575 flags.go:64] FLAG: --event-qps="50" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910953 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910956 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910960 2575 flags.go:64] FLAG: --eviction-hard="" Jul 26 17:49:29.916694 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910964 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910967 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910970 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910973 2575 flags.go:64] FLAG: --eviction-soft="" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910976 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910979 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910982 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910985 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910988 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910991 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910994 2575 flags.go:64] FLAG: --feature-gates="" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.910998 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911003 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911007 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911010 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911014 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911017 2575 flags.go:64] FLAG: --help="false" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911020 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-219.ec2.internal" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911023 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911026 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911029 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911032 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911036 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911039 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 17:49:29.917290 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911042 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911045 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911047 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911051 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911054 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911057 2575 flags.go:64] FLAG: --kube-reserved="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911059 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911062 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911065 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911068 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911072 2575 flags.go:64] FLAG: --lock-file="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911075 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911078 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911081 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911087 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911090 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911092 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911095 2575 flags.go:64] FLAG: --logging-format="text" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911098 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911102 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911106 2575 flags.go:64] FLAG: --manifest-url="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911109 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911114 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911117 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911122 2575 flags.go:64] FLAG: --max-pods="110" Jul 26 17:49:29.917909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911125 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911129 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911131 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911134 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911137 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911140 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911143 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911152 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911155 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911157 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911161 2575 flags.go:64] FLAG: --pod-cidr="" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911163 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911169 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911172 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911176 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911179 2575 flags.go:64] FLAG: --port="10250" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911182 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911185 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6b439f3113e0646" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911188 2575 flags.go:64] FLAG: --qos-reserved="" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911191 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911194 2575 flags.go:64] FLAG: --register-node="true" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911197 2575 flags.go:64] FLAG: --register-with-taints="" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911200 2575 flags.go:64] FLAG: --registry-burst="10" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911203 2575 flags.go:64] FLAG: --registry-qps="5" Jul 26 17:49:29.918549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911206 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911209 2575 flags.go:64] FLAG: --reserved-memory="" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911213 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911220 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911223 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911227 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911230 2575 flags.go:64] FLAG: --runonce="false" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911233 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911237 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911240 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911243 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911246 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911249 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911252 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911256 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911258 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911262 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911264 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911267 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911270 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911273 2575 flags.go:64] FLAG: --system-cgroups="" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911276 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911282 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911285 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911288 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 17:49:29.919138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911292 2575 flags.go:64] FLAG: --tls-min-version="" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911295 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911298 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911300 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911303 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911306 2575 flags.go:64] FLAG: --v="2" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911310 2575 flags.go:64] FLAG: --version="false" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911314 2575 flags.go:64] FLAG: --vmodule="" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911318 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.911321 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911447 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911451 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911454 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911457 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911460 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911462 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911466 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911468 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911471 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911474 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911476 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.919743 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911479 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911483 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911488 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911492 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911495 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911498 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911501 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911504 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911506 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911509 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911511 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911514 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911517 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911519 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911522 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911524 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911527 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911529 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911532 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911535 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.920275 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911537 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911543 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911546 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911549 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911551 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911554 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911557 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911559 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911562 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911565 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911568 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911570 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911573 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911576 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911579 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911582 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911584 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911587 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911589 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911592 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.920764 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911595 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911597 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911599 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911602 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911605 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911607 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911610 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911612 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911615 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911617 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911620 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911622 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911626 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911631 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911634 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911637 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911640 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911643 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911646 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.921292 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911648 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911651 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911654 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911657 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911660 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911662 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911665 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911667 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911670 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911673 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911676 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911678 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911681 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911683 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911686 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911688 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911691 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911693 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911696 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911698 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.921771 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911701 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911704 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911706 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911709 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911711 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911714 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911719 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911722 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.911724 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.912359 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.919082 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.919103 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.920354 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.920361 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 17:49:29.922278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920422 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920430 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920434 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920437 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920440 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920443 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920446 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920448 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920451 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920454 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920457 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920460 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920462 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920465 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920468 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920471 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920473 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920476 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920478 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920481 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.922628 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920484 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920487 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920489 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920492 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920494 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920498 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920503 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920508 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920512 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920522 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920527 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920531 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920533 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920536 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920539 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920541 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920544 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920546 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920549 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920551 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.923127 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920554 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920556 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920559 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920561 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920564 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920567 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920569 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920572 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920574 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920577 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920579 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920582 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920585 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920587 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920590 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920593 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920596 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920599 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920601 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920604 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.923610 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920606 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920609 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920613 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920615 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920618 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920620 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920623 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920626 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920629 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920631 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920634 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920637 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920640 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920642 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920645 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920647 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920650 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920653 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920655 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.924278 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920658 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920660 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920663 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920665 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920668 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920671 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920676 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920679 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920682 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920685 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920688 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920691 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920694 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920697 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920699 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920702 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920705 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920708 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920711 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.924737 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920713 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.920718 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920854 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920859 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920862 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920865 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920868 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920871 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920874 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920876 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920879 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920882 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920885 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920887 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920890 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 17:49:29.925220 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920893 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920895 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920898 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920901 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920904 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920906 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920909 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920911 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920914 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920916 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920919 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920922 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920924 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920927 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920930 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920933 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920935 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920938 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920940 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920943 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 17:49:29.925587 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920945 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920948 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920950 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920953 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920955 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920958 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920961 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920963 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920966 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920968 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920971 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920974 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920976 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920979 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920982 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920985 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920988 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920990 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920994 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.920997 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 17:49:29.926089 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921000 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921003 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921006 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921009 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921011 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921014 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921017 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921019 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921022 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921025 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921027 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921030 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921032 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921034 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921037 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921039 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921042 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921044 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921047 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921049 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 17:49:29.926574 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921052 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921055 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921058 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921062 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921065 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921067 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921070 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921073 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921075 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921078 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921081 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921083 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921086 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921088 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921091 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921094 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921096 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921099 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921101 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 17:49:29.927056 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921104 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921106 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921109 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921112 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921114 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921117 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:29.921119 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.921125 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.921248 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.923245 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.924610 2575 server.go:1019] "Starting client certificate rotation" Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.924735 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 17:49:29.927540 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.924808 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 17:49:29.931814 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.931777 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 17:49:29.932962 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.932945 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 17:49:29.939546 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.939528 2575 log.go:25] "Validated CRI v1 runtime API" Jul 26 17:49:29.941456 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.941440 2575 log.go:25] "Validated CRI v1 image API" Jul 26 17:49:29.942975 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.942961 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 17:49:29.948622 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.948604 2575 fs.go:135] Filesystem UUIDs: map[34b919b9-4236-42c7-a01d-7d1bfcb29e7e:/dev/nvme0n1p3 40b66301-30e4-49a2-af97-ef4fade66c40:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 26 17:49:29.948689 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.948622 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 17:49:29.954561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.954414 2575 manager.go:216] Machine: {Timestamp:2026-07-26 17:49:29.952193996 +0000 UTC m=+0.351223031 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121890 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c6f499e8d7118743fa4499bece06 SystemUUID:ec20c6f4-99e8-d711-8743-fa4499bece06 BootID:ccdf3b58-72c0-40f2-9125-9087527ea76f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6f:b9:7f:fa:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6f:b9:7f:fa:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:7e:d5:4a:77:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 17:49:29.954561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.954558 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 17:49:29.954666 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.954640 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 17:49:29.958294 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.958261 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 17:49:29.958448 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.958297 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 17:49:29.959086 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.959077 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 17:49:29.959117 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.959089 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 17:49:29.959117 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.959096 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 17:49:29.959851 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.959836 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 17:49:29.959919 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.959897 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 17:49:29.960667 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.960657 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 17:49:29.963337 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963324 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 26 17:49:29.963485 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963475 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 17:49:29.963522 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963512 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 26 17:49:29.963553 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963543 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 17:49:29.963585 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963563 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 17:49:29.963619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963593 2575 kubelet.go:398] "Adding apiserver pod source" Jul 26 17:49:29.963619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.963603 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 17:49:29.964141 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.964124 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 17:49:29.964219 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.964152 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 17:49:29.967628 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.967597 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 17:49:29.971081 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.971062 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 17:49:29.971148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.971117 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 17:49:29.972784 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972766 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972788 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972799 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972820 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972833 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972842 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972854 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972859 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972867 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 17:49:29.972860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972873 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 17:49:29.973283 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972883 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 17:49:29.973283 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.972888 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 17:49:29.974459 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.974446 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 17:49:29.974459 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.974459 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 17:49:29.975805 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.975782 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 17:49:29.975805 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.975786 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 17:49:29.978046 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.978032 2575 server.go:1268] "Started kubelet" Jul 26 17:49:29.978148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.978117 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 17:49:29.978203 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.978120 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 17:49:29.978251 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.978229 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 17:49:29.979080 ip-10-0-138-219 systemd[1]: Started Kubernetes Kubelet. Jul 26 17:49:29.979983 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.979851 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 17:49:29.980395 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.980378 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 26 17:49:29.983645 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.983624 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 17:49:29.984609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.984589 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 17:49:29.985235 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.985217 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j97bz" Jul 26 17:49:29.985848 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.984387 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-219.ec2.internal.18c5e83b984a7548 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-219.ec2.internal,UID:ip-10-0-138-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-219.ec2.internal,},FirstTimestamp:2026-07-26 17:49:29.978000712 +0000 UTC m=+0.377029750,LastTimestamp:2026-07-26 17:49:29.978000712 +0000 UTC m=+0.377029750,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-219.ec2.internal,}" Jul 26 17:49:29.986766 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.986744 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 17:49:29.986766 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.986750 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 17:49:29.987077 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.987054 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:29.987463 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.987441 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 17:49:29.987583 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.987455 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 17:49:29.987684 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.987591 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 17:49:29.987751 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.987709 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 26 17:49:29.987751 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.987719 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 26 17:49:29.988056 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988041 2575 factory.go:153] Registering CRI-O factory Jul 26 17:49:29.988120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988061 2575 factory.go:223] Registration of the crio container factory successfully Jul 26 17:49:29.988120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988112 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 17:49:29.988243 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988122 2575 factory.go:55] Registering systemd factory Jul 26 17:49:29.988243 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988130 2575 factory.go:223] Registration of the systemd container factory successfully Jul 26 17:49:29.988243 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988154 2575 factory.go:103] Registering Raw factory Jul 26 17:49:29.988243 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988168 2575 manager.go:1196] Started watching for new ooms in manager Jul 26 17:49:29.988836 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.988820 2575 manager.go:349] Starting recovery of all containers Jul 26 17:49:29.993843 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.993819 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 26 17:49:29.993972 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:29.993845 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 26 17:49:29.994428 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:29.994405 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j97bz" Jul 26 17:49:30.001473 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.001312 2575 manager.go:354] Recovery completed Jul 26 17:49:30.006273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.006258 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.008570 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.008547 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.008644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.008583 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.008644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.008598 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.009187 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.009172 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 17:49:30.009187 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.009185 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 17:49:30.009305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.009189 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.009305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.009200 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 26 17:49:30.009997 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.009982 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.010040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.010004 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.010040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.010015 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.010943 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.010862 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-219.ec2.internal.18c5e83b9a1cda6a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-219.ec2.internal,UID:ip-10-0-138-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-219.ec2.internal,},FirstTimestamp:2026-07-26 17:49:30.008566378 +0000 UTC m=+0.407595416,LastTimestamp:2026-07-26 17:49:30.008566378 +0000 UTC m=+0.407595416,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-219.ec2.internal,}" Jul 26 17:49:30.012040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.012028 2575 policy_none.go:49] "None policy: Start" Jul 26 17:49:30.012124 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.012045 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 17:49:30.012124 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.012056 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 17:49:30.014134 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.014119 2575 policy_none.go:47] "Start" Jul 26 17:49:30.057465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057442 2575 manager.go:337] "Starting Device Plugin manager" Jul 26 17:49:30.057559 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.057483 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 17:49:30.057559 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057498 2575 server.go:87] "Starting device plugin registration server" Jul 26 17:49:30.057802 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057790 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 17:49:30.057869 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057805 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 17:49:30.057922 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057906 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 17:49:30.058012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.057991 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 17:49:30.058012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.058005 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 17:49:30.058478 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.058460 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 17:49:30.058574 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.058506 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.158601 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.158521 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.160914 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.159530 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.160914 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.159560 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.160914 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.159571 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.160914 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.159607 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.168285 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.168263 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.168393 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.168296 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-219.ec2.internal\": node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.186235 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.186207 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.200492 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.200446 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 17:49:30.201836 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.201817 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 17:49:30.201947 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.201850 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 17:49:30.201947 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.201874 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 17:49:30.201947 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.201921 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 17:49:30.205356 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.205334 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:30.286955 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.286917 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.302156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.302120 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"] Jul 26 17:49:30.302226 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.302205 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.303935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.303917 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.304043 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.303957 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.304043 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.303972 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.306519 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.306499 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.306639 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.306620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.306691 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.306665 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.307411 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307394 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.307509 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307394 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.307509 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307448 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.307509 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307462 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.307509 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307425 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.307509 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.307505 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.309881 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.309866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.309951 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.309896 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 17:49:30.312596 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.312578 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientMemory" Jul 26 17:49:30.312667 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.312616 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 17:49:30.312667 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.312633 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeHasSufficientPID" Jul 26 17:49:30.333767 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.333743 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-219.ec2.internal\" not found" node="ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.338996 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.338975 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-219.ec2.internal\" not found" node="ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.387055 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.387019 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.389433 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.389414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.389485 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.389443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1de7be92a0a914801205d3ece54cca0e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"1de7be92a0a914801205d3ece54cca0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.389485 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.389461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.488168 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.488093 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.490546 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.490631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.490631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1de7be92a0a914801205d3ece54cca0e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"1de7be92a0a914801205d3ece54cca0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.490730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.490730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490630 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3deac22f55b27978c45b7ec5b962b9ed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal\" (UID: \"3deac22f55b27978c45b7ec5b962b9ed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.490730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.490631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1de7be92a0a914801205d3ece54cca0e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-219.ec2.internal\" (UID: \"1de7be92a0a914801205d3ece54cca0e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.589109 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.589069 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.639219 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.639176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.642503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.642476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:30.689786 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.689755 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.791945 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.791861 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.831020 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.830990 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:30.893078 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.893042 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:30.924206 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.924168 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 17:49:30.924830 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.924391 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 17:49:30.924830 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.924394 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 17:49:30.992880 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:30.992852 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:30.993938 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:30.993912 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:31.000845 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.000818 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 17:44:30 +0000 UTC" deadline="2028-03-06 08:55:27.691917652 +0000 UTC" Jul 26 17:49:31.000845 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.000845 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14127h5m56.691076016s" Jul 26 17:49:31.094896 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:31.094824 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-219.ec2.internal\" not found" Jul 26 17:49:31.149441 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.149408 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:31.188019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.187982 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" Jul 26 17:49:31.200300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.200275 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 17:49:31.201505 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.201490 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" Jul 26 17:49:31.208776 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.208753 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 17:49:31.240932 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:31.240885 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deac22f55b27978c45b7ec5b962b9ed.slice/crio-caa6bc1d6aa77c49dd3f315fc752be709ad76d25e8dae3bb8fe4751fb4961614 WatchSource:0}: Error finding container caa6bc1d6aa77c49dd3f315fc752be709ad76d25e8dae3bb8fe4751fb4961614: Status 404 returned error can't find the container with id caa6bc1d6aa77c49dd3f315fc752be709ad76d25e8dae3bb8fe4751fb4961614 Jul 26 17:49:31.241436 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:31.241412 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de7be92a0a914801205d3ece54cca0e.slice/crio-6b14378367d814d45921216eaca018aa74de43628400592e14d580d9b3d56830 WatchSource:0}: Error finding container 6b14378367d814d45921216eaca018aa74de43628400592e14d580d9b3d56830: Status 404 returned error can't find the container with id 6b14378367d814d45921216eaca018aa74de43628400592e14d580d9b3d56830 Jul 26 17:49:31.246141 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.246122 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 17:49:31.964586 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.964552 2575 apiserver.go:52] "Watching apiserver" Jul 26 17:49:31.977891 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.977853 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 17:49:31.979266 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.979231 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-jpchf","openshift-multus/multus-additional-cni-plugins-2lr4f","openshift-multus/multus-mj8vn","openshift-multus/network-metrics-daemon-7lx5k","openshift-ovn-kubernetes/ovnkube-node-5qwpk","kube-system/global-pull-secret-syncer-6dpt6","kube-system/konnectivity-agent-g9tw7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq","openshift-dns/node-resolver-fqjvs","openshift-image-registry/node-ca-dbvcm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal","openshift-network-diagnostics/network-check-target-xq9pn","openshift-network-operator/iptables-alerter-k6vm5","kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal"] Jul 26 17:49:31.982037 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.982011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:31.985145 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.985120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:31.985386 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.985366 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 17:49:31.985574 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.985556 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 17:49:31.985977 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.985958 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wfdmr\"" Jul 26 17:49:31.987936 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.987920 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 17:49:31.988545 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.988522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mj8vn" Jul 26 17:49:31.989321 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.989301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ffzbq\"" Jul 26 17:49:31.989321 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.989316 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 17:49:31.989459 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.989442 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 17:49:31.989608 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.989579 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 17:49:31.989876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.989818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:31.989977 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:31.989895 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:31.991366 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.991125 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 17:49:31.991366 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.991172 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 17:49:31.992107 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.992083 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 17:49:31.992370 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.992352 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-22gg5\"" Jul 26 17:49:31.992618 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.992598 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 17:49:31.992968 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.992945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:31.995550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.995531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:31.995623 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:31.995600 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:31.996333 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.996310 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 17:49:31.996670 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.996650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 17:49:31.996908 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.996889 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 17:49:31.997088 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.997069 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 17:49:31.997686 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.997667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ggpm7\"" Jul 26 17:49:31.997897 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.997880 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 17:49:31.998166 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.998142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:31.998254 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:31.998192 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 17:49:32.000082 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.000062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.000855 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.000782 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 17:49:32.001182 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.001160 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 17:49:32.001376 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.001163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d66l6\"" Jul 26 17:49:32.002231 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-netns\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-kubelet\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc558bc8-71a8-4a1a-9144-e3325e963098-host-slash\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-socket-dir-parent\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-hostroot\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-multus-certs\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsk89\" (UniqueName: \"kubernetes.io/projected/55beaa58-5db7-4727-a49b-ef29621b3dcb-kube-api-access-rsk89\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-agent-certs\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc558bc8-71a8-4a1a-9144-e3325e963098-iptables-alerter-script\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76v6\" (UniqueName: \"kubernetes.io/projected/fc558bc8-71a8-4a1a-9144-e3325e963098-kube-api-access-c76v6\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-bin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-konnectivity-ca\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-k8s-cni-cncf-io\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-multus\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-conf-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002704 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 17:49:32.002944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-daemon-config\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.002977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-etc-kubernetes\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwgj\" (UniqueName: \"kubernetes.io/projected/fa10fb0a-6825-48da-9850-e455f99ca78b-kube-api-access-mcwgj\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-system-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-cnibin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003116 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cn228\"" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-os-release\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-cni-binary-copy\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003406 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 17:49:32.003479 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003475 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 17:49:32.004111 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.003732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.005565 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.005537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.007231 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.007211 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 17:49:32.007347 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.007330 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5szc2\"" Jul 26 17:49:32.007442 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.007428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 17:49:32.008972 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.008953 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 17:49:32.008972 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.008968 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 17:49:32.009187 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.009164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.009276 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.009199 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 17:49:32.009276 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.009207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cp27q\"" Jul 26 17:49:32.011382 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.011105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:32.011382 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.011164 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:32.011788 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.011760 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 17:49:32.012140 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.012118 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2x284\"" Jul 26 17:49:32.012301 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.012277 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 17:49:32.012571 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.012531 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 17:49:32.029661 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.028046 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7xhbv" Jul 26 17:49:32.035428 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.034953 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7xhbv" Jul 26 17:49:32.088796 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.088723 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 17:49:32.104048 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz665\" (UniqueName: \"kubernetes.io/projected/430af05c-ecbe-4796-b18b-8c11872a3d84-kube-api-access-wz665\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-sys-fs\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-konnectivity-ca\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-os-release\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223047b2-a308-42a1-b28c-5be77c548d7a-hosts-file\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.104213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-cni-binary-copy\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cnibin\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsk89\" (UniqueName: \"kubernetes.io/projected/55beaa58-5db7-4727-a49b-ef29621b3dcb-kube-api-access-rsk89\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-host\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-system-cni-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-etc-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-env-overrides\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-bin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.104528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-systemd\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-ovn\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-etc-kubernetes\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-kubernetes\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-netd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxf9\" (UniqueName: \"kubernetes.io/projected/223047b2-a308-42a1-b28c-5be77c548d7a-kube-api-access-tsxf9\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-system-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-os-release\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-etc-tuned\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcbx\" (UniqueName: \"kubernetes.io/projected/f41c2f6a-1953-4d57-8c13-475c8b635c5c-kube-api-access-ftcbx\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-bin\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-etc-kubernetes\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.104946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-script-lib\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-os-release\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-bin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jr9\" (UniqueName: \"kubernetes.io/projected/ee547b8b-b19d-440c-8d61-8297014e9e5a-kube-api-access-m8jr9\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-hostroot\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.105211 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-dbus\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.105304 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:49:32.605263369 +0000 UTC m=+3.004292401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-hostroot\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-konnectivity-ca\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-agent-certs\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-system-cni-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-daemon-config\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-var-lib-kubelet\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430af05c-ecbe-4796-b18b-8c11872a3d84-host\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105625 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-cni-binary-copy\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.105832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-systemd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-log-socket\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105783 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-k8s-cni-cncf-io\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-conf-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430af05c-ecbe-4796-b18b-8c11872a3d84-serviceca\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-device-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-k8s-cni-cncf-io\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcf8\" (UniqueName: \"kubernetes.io/projected/b9f61d50-93d8-4424-b167-471aa98e826b-kube-api-access-zdcf8\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105967 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-conf-dir\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.105985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-var-lib-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-config\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-daemon-config\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-kubelet-config\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-cnibin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-kubelet\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.106717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-kubelet\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-cnibin\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-kubelet\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-slash\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-netns\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-multus-certs\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-multus-certs\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-lib-modules\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-systemd-units\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc558bc8-71a8-4a1a-9144-e3325e963098-iptables-alerter-script\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76v6\" (UniqueName: \"kubernetes.io/projected/fc558bc8-71a8-4a1a-9144-e3325e963098-kube-api-access-c76v6\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwgj\" (UniqueName: \"kubernetes.io/projected/fa10fb0a-6825-48da-9850-e455f99ca78b-kube-api-access-mcwgj\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-modprobe-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-run\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-tmp\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.107484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-registration-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-multus\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-socket-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-node-log\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223047b2-a308-42a1-b28c-5be77c548d7a-tmp-dir\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-netns\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-conf\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bn6x\" (UniqueName: \"kubernetes.io/projected/7db03598-347b-4241-9167-b276229e26b4-kube-api-access-5bn6x\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc558bc8-71a8-4a1a-9144-e3325e963098-host-slash\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.106992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-socket-dir-parent\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysconfig\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-sys\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fc558bc8-71a8-4a1a-9144-e3325e963098-iptables-alerter-script\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-run-netns\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.108300 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-multus-socket-dir-parent\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.109151 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc558bc8-71a8-4a1a-9144-e3325e963098-host-slash\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.109151 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.107387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55beaa58-5db7-4727-a49b-ef29621b3dcb-host-var-lib-cni-multus\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.109811 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.109785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e879f3c-533b-44ca-bb2e-aafe77e9c9d1-agent-certs\") pod \"konnectivity-agent-g9tw7\" (UID: \"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1\") " pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.115263 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.115193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsk89\" (UniqueName: \"kubernetes.io/projected/55beaa58-5db7-4727-a49b-ef29621b3dcb-kube-api-access-rsk89\") pod \"multus-mj8vn\" (UID: \"55beaa58-5db7-4727-a49b-ef29621b3dcb\") " pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.119356 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.119336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76v6\" (UniqueName: \"kubernetes.io/projected/fc558bc8-71a8-4a1a-9144-e3325e963098-kube-api-access-c76v6\") pod \"iptables-alerter-k6vm5\" (UID: \"fc558bc8-71a8-4a1a-9144-e3325e963098\") " pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.119494 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.119470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwgj\" (UniqueName: \"kubernetes.io/projected/fa10fb0a-6825-48da-9850-e455f99ca78b-kube-api-access-mcwgj\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.207763 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.207954 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-kubelet\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.207954 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-slash\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.207954 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-netns\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.207954 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-lib-modules\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.207954 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-systemd-units\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.207988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-modprobe-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-run\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-tmp\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-registration-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-socket-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-node-log\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.208229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223047b2-a308-42a1-b28c-5be77c548d7a-tmp-dir\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-conf\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bn6x\" (UniqueName: \"kubernetes.io/projected/7db03598-347b-4241-9167-b276229e26b4-kube-api-access-5bn6x\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysconfig\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-sys\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz665\" (UniqueName: \"kubernetes.io/projected/430af05c-ecbe-4796-b18b-8c11872a3d84-kube-api-access-wz665\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-sys-fs\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-os-release\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223047b2-a308-42a1-b28c-5be77c548d7a-hosts-file\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cnibin\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.208714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-host\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-system-cni-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-etc-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-env-overrides\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-systemd\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-ovn\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.208984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-kubernetes\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-netd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxf9\" (UniqueName: \"kubernetes.io/projected/223047b2-a308-42a1-b28c-5be77c548d7a-kube-api-access-tsxf9\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-etc-tuned\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcbx\" (UniqueName: \"kubernetes.io/projected/f41c2f6a-1953-4d57-8c13-475c8b635c5c-kube-api-access-ftcbx\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-bin\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-script-lib\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.209319 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jr9\" (UniqueName: \"kubernetes.io/projected/ee547b8b-b19d-440c-8d61-8297014e9e5a-kube-api-access-m8jr9\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-dbus\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-var-lib-kubelet\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430af05c-ecbe-4796-b18b-8c11872a3d84-host\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-systemd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-log-socket\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430af05c-ecbe-4796-b18b-8c11872a3d84-serviceca\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-device-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcf8\" (UniqueName: \"kubernetes.io/projected/b9f61d50-93d8-4424-b167-471aa98e826b-kube-api-access-zdcf8\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-var-lib-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-config\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-kubelet-config\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-host\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-kubelet\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.210112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209854 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-slash\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-kubelet-config\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-netns\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-lib-modules\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.209981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-system-cni-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-systemd-units\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.210065 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-etc-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.210131 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:32.710116703 +0000 UTC m=+3.109145728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-modprobe-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-run\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-env-overrides\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.210940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-systemd\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.211010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-ovn\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.211644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-registration-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.211644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211486 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-socket-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.211644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211545 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-node-log\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.211909 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-kubernetes\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.211960 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223047b2-a308-42a1-b28c-5be77c548d7a-tmp-dir\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.211960 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.211928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-netd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.212128 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.212104 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-conf\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.212940 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.212918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.213063 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.212951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.213063 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213030 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysconfig\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.213168 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-sys\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.213168 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-run-systemd\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.213168 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-tmp\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.213168 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223047b2-a308-42a1-b28c-5be77c548d7a-hosts-file\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-cni-bin\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-var-lib-openvswitch\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-os-release\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-etc-sysctl-d\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cnibin\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.213361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-device-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.213741 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-log-socket\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.213741 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb57ba8d-0081-4e01-baec-552dfcc34995-dbus\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.213741 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-sys-fs\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.213741 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/430af05c-ecbe-4796-b18b-8c11872a3d84-host\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.213741 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7db03598-347b-4241-9167-b276229e26b4-var-lib-kubelet\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.214015 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.213970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f61d50-93d8-4424-b167-471aa98e826b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.214153 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee547b8b-b19d-440c-8d61-8297014e9e5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.214354 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/430af05c-ecbe-4796-b18b-8c11872a3d84-serviceca\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.214441 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovn-node-metrics-cert\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.214525 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-config\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.214572 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee547b8b-b19d-440c-8d61-8297014e9e5a-ovnkube-script-lib\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.214935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.214847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f41c2f6a-1953-4d57-8c13-475c8b635c5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.216229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.216204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7db03598-347b-4241-9167-b276229e26b4-etc-tuned\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.217450 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.217261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" event={"ID":"1de7be92a0a914801205d3ece54cca0e","Type":"ContainerStarted","Data":"6b14378367d814d45921216eaca018aa74de43628400592e14d580d9b3d56830"} Jul 26 17:49:32.219465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.219436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"3deac22f55b27978c45b7ec5b962b9ed","Type":"ContainerStarted","Data":"caa6bc1d6aa77c49dd3f315fc752be709ad76d25e8dae3bb8fe4751fb4961614"} Jul 26 17:49:32.220215 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.220186 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:32.220323 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.220218 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:32.220323 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.220235 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:32.220323 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.220288 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:32.720274355 +0000 UTC m=+3.119303380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:32.227826 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.227755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxf9\" (UniqueName: \"kubernetes.io/projected/223047b2-a308-42a1-b28c-5be77c548d7a-kube-api-access-tsxf9\") pod \"node-resolver-fqjvs\" (UID: \"223047b2-a308-42a1-b28c-5be77c548d7a\") " pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.227972 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.227825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bn6x\" (UniqueName: \"kubernetes.io/projected/7db03598-347b-4241-9167-b276229e26b4-kube-api-access-5bn6x\") pod \"tuned-jpchf\" (UID: \"7db03598-347b-4241-9167-b276229e26b4\") " pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.229368 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.229273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcbx\" (UniqueName: \"kubernetes.io/projected/f41c2f6a-1953-4d57-8c13-475c8b635c5c-kube-api-access-ftcbx\") pod \"multus-additional-cni-plugins-2lr4f\" (UID: \"f41c2f6a-1953-4d57-8c13-475c8b635c5c\") " pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.229604 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.229554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jr9\" (UniqueName: \"kubernetes.io/projected/ee547b8b-b19d-440c-8d61-8297014e9e5a-kube-api-access-m8jr9\") pod \"ovnkube-node-5qwpk\" (UID: \"ee547b8b-b19d-440c-8d61-8297014e9e5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.231637 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.231591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz665\" (UniqueName: \"kubernetes.io/projected/430af05c-ecbe-4796-b18b-8c11872a3d84-kube-api-access-wz665\") pod \"node-ca-dbvcm\" (UID: \"430af05c-ecbe-4796-b18b-8c11872a3d84\") " pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.232023 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.232002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcf8\" (UniqueName: \"kubernetes.io/projected/b9f61d50-93d8-4424-b167-471aa98e826b-kube-api-access-zdcf8\") pod \"aws-ebs-csi-driver-node-6qllq\" (UID: \"b9f61d50-93d8-4424-b167-471aa98e826b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.301635 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.301598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:32.313136 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.313110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6vm5" Jul 26 17:49:32.323325 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.323298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mj8vn" Jul 26 17:49:32.330484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.330456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:32.338582 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.338557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqjvs" Jul 26 17:49:32.346485 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.346462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpchf" Jul 26 17:49:32.354400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.354379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" Jul 26 17:49:32.364336 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.364305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dbvcm" Jul 26 17:49:32.370310 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.370291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" Jul 26 17:49:32.431460 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.431431 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:32.613191 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.613097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:32.613364 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.613252 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:32.613364 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.613320 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:49:33.613300897 +0000 UTC m=+4.012329920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:32.713635 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.713583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:32.713836 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.713766 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:32.713900 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.713851 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:33.713834775 +0000 UTC m=+4.112863799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:32.755361 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.755317 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 17:49:32.814447 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:32.814403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:32.814643 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.814576 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:32.814643 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.814599 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:32.814643 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.814610 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:32.814847 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:32.814678 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:33.814663515 +0000 UTC m=+4.213692562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:33.038636 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.038548 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 17:44:32 +0000 UTC" deadline="2028-03-21 06:47:30.625847217 +0000 UTC" Jul 26 17:49:33.038636 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.038586 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h57m57.587266904s" Jul 26 17:49:33.203138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.203105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:33.203318 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.203109 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:33.203318 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.203250 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:33.203432 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.203365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:33.248291 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.248256 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e879f3c_533b_44ca_bb2e_aafe77e9c9d1.slice/crio-c476404e48ff39478b4bab9e5f7a527087019ecb9a424bbfb22f26907e447028 WatchSource:0}: Error finding container c476404e48ff39478b4bab9e5f7a527087019ecb9a424bbfb22f26907e447028: Status 404 returned error can't find the container with id c476404e48ff39478b4bab9e5f7a527087019ecb9a424bbfb22f26907e447028 Jul 26 17:49:33.249481 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.249419 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc558bc8_71a8_4a1a_9144_e3325e963098.slice/crio-86b9782ee86ce3f98f2e17ca509a50fb36dff3aa5d56a9847ed9d4f30474dacd WatchSource:0}: Error finding container 86b9782ee86ce3f98f2e17ca509a50fb36dff3aa5d56a9847ed9d4f30474dacd: Status 404 returned error can't find the container with id 86b9782ee86ce3f98f2e17ca509a50fb36dff3aa5d56a9847ed9d4f30474dacd Jul 26 17:49:33.251826 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.251784 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db03598_347b_4241_9167_b276229e26b4.slice/crio-992c804c41d842459a611e868f0d1002d87d62dc84010d12b031d0d90792fa78 WatchSource:0}: Error finding container 992c804c41d842459a611e868f0d1002d87d62dc84010d12b031d0d90792fa78: Status 404 returned error can't find the container with id 992c804c41d842459a611e868f0d1002d87d62dc84010d12b031d0d90792fa78 Jul 26 17:49:33.254997 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.254970 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41c2f6a_1953_4d57_8c13_475c8b635c5c.slice/crio-dbb6766a856838189e2b17b1a3d6c40a6b66d13887dac59685e003b967937ddd WatchSource:0}: Error finding container dbb6766a856838189e2b17b1a3d6c40a6b66d13887dac59685e003b967937ddd: Status 404 returned error can't find the container with id dbb6766a856838189e2b17b1a3d6c40a6b66d13887dac59685e003b967937ddd Jul 26 17:49:33.257930 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.257728 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223047b2_a308_42a1_b28c_5be77c548d7a.slice/crio-625cf5ed0d88f1f93a70b471621893c801b863a6e13ded5dd268c57ca208ab42 WatchSource:0}: Error finding container 625cf5ed0d88f1f93a70b471621893c801b863a6e13ded5dd268c57ca208ab42: Status 404 returned error can't find the container with id 625cf5ed0d88f1f93a70b471621893c801b863a6e13ded5dd268c57ca208ab42 Jul 26 17:49:33.258150 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.258119 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f61d50_93d8_4424_b167_471aa98e826b.slice/crio-0e507beb288da555c615ecb0b7d922e963fa38e0fe2eeb4df0a862a7d7571eb0 WatchSource:0}: Error finding container 0e507beb288da555c615ecb0b7d922e963fa38e0fe2eeb4df0a862a7d7571eb0: Status 404 returned error can't find the container with id 0e507beb288da555c615ecb0b7d922e963fa38e0fe2eeb4df0a862a7d7571eb0 Jul 26 17:49:33.258942 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.258880 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430af05c_ecbe_4796_b18b_8c11872a3d84.slice/crio-b384844726c41477cb1036b27d48b0fd5ba9c1a85e18a188d2e9dc3aa1842fe6 WatchSource:0}: Error finding container b384844726c41477cb1036b27d48b0fd5ba9c1a85e18a188d2e9dc3aa1842fe6: Status 404 returned error can't find the container with id b384844726c41477cb1036b27d48b0fd5ba9c1a85e18a188d2e9dc3aa1842fe6 Jul 26 17:49:33.260979 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:49:33.260883 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55beaa58_5db7_4727_a49b_ef29621b3dcb.slice/crio-871c95c3ab93ebe4420f6ac2bcbb4c7a37d09ae1477fabe332f116221e57b702 WatchSource:0}: Error finding container 871c95c3ab93ebe4420f6ac2bcbb4c7a37d09ae1477fabe332f116221e57b702: Status 404 returned error can't find the container with id 871c95c3ab93ebe4420f6ac2bcbb4c7a37d09ae1477fabe332f116221e57b702 Jul 26 17:49:33.620849 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.620523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:33.620849 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.620716 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:33.621107 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.620857 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:49:35.620818598 +0000 UTC m=+6.019847627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:33.722370 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.721727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:33.722370 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.721923 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:33.722370 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.721998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:35.721976163 +0000 UTC m=+6.121005204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:33.823928 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:33.823140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:33.823928 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.823352 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:33.823928 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.823374 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:33.823928 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.823388 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:33.823928 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:33.823451 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:35.823430357 +0000 UTC m=+6.222459385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:34.039326 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.039250 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 17:44:32 +0000 UTC" deadline="2028-01-10 11:52:46.765102077 +0000 UTC" Jul 26 17:49:34.039326 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.039292 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12786h3m12.72581535s" Jul 26 17:49:34.208494 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.208405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:34.208658 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:34.208555 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:34.249638 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.249599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" event={"ID":"1de7be92a0a914801205d3ece54cca0e","Type":"ContainerStarted","Data":"479e4668efbb8cc12999234e29f86a9a460bfdd1d14fc8e5dee11ca707d7c5f1"} Jul 26 17:49:34.254243 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.254201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mj8vn" event={"ID":"55beaa58-5db7-4727-a49b-ef29621b3dcb","Type":"ContainerStarted","Data":"871c95c3ab93ebe4420f6ac2bcbb4c7a37d09ae1477fabe332f116221e57b702"} Jul 26 17:49:34.274358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.274316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbvcm" event={"ID":"430af05c-ecbe-4796-b18b-8c11872a3d84","Type":"ContainerStarted","Data":"b384844726c41477cb1036b27d48b0fd5ba9c1a85e18a188d2e9dc3aa1842fe6"} Jul 26 17:49:34.285832 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.285763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqjvs" event={"ID":"223047b2-a308-42a1-b28c-5be77c548d7a","Type":"ContainerStarted","Data":"625cf5ed0d88f1f93a70b471621893c801b863a6e13ded5dd268c57ca208ab42"} Jul 26 17:49:34.298811 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.298725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerStarted","Data":"dbb6766a856838189e2b17b1a3d6c40a6b66d13887dac59685e003b967937ddd"} Jul 26 17:49:34.312188 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.312007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpchf" event={"ID":"7db03598-347b-4241-9167-b276229e26b4","Type":"ContainerStarted","Data":"992c804c41d842459a611e868f0d1002d87d62dc84010d12b031d0d90792fa78"} Jul 26 17:49:34.330876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.330815 2575 generic.go:356] "Generic (PLEG): container finished" podID="3deac22f55b27978c45b7ec5b962b9ed" containerID="1e0c2e98e3788064d8f45a0696bbb1b3f4d838b6a3305ac9666a1cd0cd544d52" exitCode=0 Jul 26 17:49:34.331091 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.330929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"3deac22f55b27978c45b7ec5b962b9ed","Type":"ContainerDied","Data":"1e0c2e98e3788064d8f45a0696bbb1b3f4d838b6a3305ac9666a1cd0cd544d52"} Jul 26 17:49:34.339019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.338888 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"957762f7ead90a73010415e18a964d64e5a03f6bfa47b407741c80e8f468868e"} Jul 26 17:49:34.346793 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.346623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" event={"ID":"b9f61d50-93d8-4424-b167-471aa98e826b","Type":"ContainerStarted","Data":"0e507beb288da555c615ecb0b7d922e963fa38e0fe2eeb4df0a862a7d7571eb0"} Jul 26 17:49:34.348213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.348152 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal" podStartSLOduration=2.348133061 podStartE2EDuration="2.348133061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:34.283142861 +0000 UTC m=+4.682171910" watchObservedRunningTime="2026-07-26 17:49:34.348133061 +0000 UTC m=+4.747162111" Jul 26 17:49:34.360090 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.360013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6vm5" event={"ID":"fc558bc8-71a8-4a1a-9144-e3325e963098","Type":"ContainerStarted","Data":"86b9782ee86ce3f98f2e17ca509a50fb36dff3aa5d56a9847ed9d4f30474dacd"} Jul 26 17:49:34.364346 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:34.364267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g9tw7" event={"ID":"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1","Type":"ContainerStarted","Data":"c476404e48ff39478b4bab9e5f7a527087019ecb9a424bbfb22f26907e447028"} Jul 26 17:49:35.203619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.202924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:35.203619 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.203065 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:35.203619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.203149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:35.203619 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.203246 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:35.405903 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.405840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" event={"ID":"3deac22f55b27978c45b7ec5b962b9ed","Type":"ContainerStarted","Data":"0d3a7713e21aa0f47385eb25de46a3453a2325c43fead226cfcbd4be4ea92308"} Jul 26 17:49:35.639328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.638605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:35.639328 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.638818 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:35.639328 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.638909 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:49:39.638887077 +0000 UTC m=+10.037916102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:35.739358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.739314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:35.739539 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.739480 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:35.739641 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.739549 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:39.739528788 +0000 UTC m=+10.138557840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:35.840679 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:35.840639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:35.840980 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.840951 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:35.840980 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.840980 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:35.841106 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.840994 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:35.841106 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:35.841055 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:39.841036689 +0000 UTC m=+10.240065716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:36.202288 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:36.202256 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:36.202486 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:36.202386 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:37.202489 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:37.202455 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:37.203061 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:37.202593 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:37.203061 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:37.202644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:37.203061 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:37.202718 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:38.203331 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:38.203144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:38.203331 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:38.203291 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:39.203731 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:39.202908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:39.203731 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.203072 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:39.203731 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:39.203575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:39.203731 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.203679 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:39.675122 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:39.675047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:39.675269 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.675214 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:39.675344 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.675293 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:49:47.675271173 +0000 UTC m=+18.074300201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:39.775482 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:39.775441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:39.775656 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.775557 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:39.775656 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.775604 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:47.775589695 +0000 UTC m=+18.174618718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:39.876268 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:39.876205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:39.876470 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.876434 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:39.876470 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.876469 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:39.876595 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.876486 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:39.876595 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:39.876561 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:49:47.876539302 +0000 UTC m=+18.275568329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:40.203761 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:40.203725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:40.204269 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:40.203833 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:41.202647 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:41.202505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:41.202867 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:41.202646 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:41.202993 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:41.202505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:41.203123 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:41.203109 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:42.202867 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:42.202677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:42.202867 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:42.202842 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:43.202842 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:43.202704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:43.203049 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:43.202900 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:43.203049 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:43.202704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:43.203049 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:43.203039 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:44.202143 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:44.202111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:44.202311 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:44.202273 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:45.202467 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:45.202434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:45.202843 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:45.202434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:45.202843 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:45.202553 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:45.202843 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:45.202625 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:46.202558 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:46.202523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:46.203045 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:46.202653 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:47.202872 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:47.202837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:47.203360 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:47.202879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:47.203360 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.202981 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:47.203611 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.203560 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:47.739245 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:47.739203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:47.739453 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.739401 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:47.739522 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.739474 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:50:03.739452747 +0000 UTC m=+34.138481771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:49:47.840055 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:47.840017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:47.840234 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.840144 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:47.840234 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.840206 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:03.840186848 +0000 UTC m=+34.239215894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:49:47.940593 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:47.940558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:47.940779 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.940693 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:49:47.940779 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.940710 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:49:47.940779 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.940719 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:47.940779 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:47.940772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:03.940758583 +0000 UTC m=+34.339787605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:49:48.202585 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:48.202507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:48.202719 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:48.202615 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:49.202503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:49.202467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:49.202960 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:49.202592 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:49.202960 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:49.202649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:49.202960 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:49.202822 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:50.203989 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:50.203950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:50.204428 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:50.204063 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:51.202420 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:51.202384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:51.202601 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:51.202540 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:51.202601 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:51.202590 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:51.202727 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:51.202683 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:52.202390 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.202357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:52.202846 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:52.202493 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:52.442610 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.442419 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mj8vn" event={"ID":"55beaa58-5db7-4727-a49b-ef29621b3dcb","Type":"ContainerStarted","Data":"8b113faeb8685e3bd32f83d4e7685e058a639e73dfe14e638e87eb8648459d4b"} Jul 26 17:49:52.444079 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.444050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dbvcm" event={"ID":"430af05c-ecbe-4796-b18b-8c11872a3d84","Type":"ContainerStarted","Data":"e845bad05fb234dabdeda7ab33c6099f7452bab1192332d7fb2f4876027e7043"} Jul 26 17:49:52.446571 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.446542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqjvs" event={"ID":"223047b2-a308-42a1-b28c-5be77c548d7a","Type":"ContainerStarted","Data":"ce35483c42aecb01098d42cec7fda97a91bea966aaca36530f71f9f124a12245"} Jul 26 17:49:52.449075 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.449044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerStarted","Data":"bc039054fdd32fd57ed6be3599d3c0ec0917929ac3cb54369757720e07aee041"} Jul 26 17:49:52.450874 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.450820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpchf" event={"ID":"7db03598-347b-4241-9167-b276229e26b4","Type":"ContainerStarted","Data":"da0900116448ef5f66708bdf9b86e285d30f9c743ef6805a2d930f356ca18b23"} Jul 26 17:49:52.452351 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.452332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"ef8f3a5b5916ee2535916dbf34a187f3d9ce4343251bab073351751713c2890d"} Jul 26 17:49:52.453999 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.453945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" event={"ID":"b9f61d50-93d8-4424-b167-471aa98e826b","Type":"ContainerStarted","Data":"e70f7d30a4db04c7b2d01dc8b83fab30e423217398fb55af24ae12b3a5aacedf"} Jul 26 17:49:52.455546 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.455504 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g9tw7" event={"ID":"8e879f3c-533b-44ca-bb2e-aafe77e9c9d1","Type":"ContainerStarted","Data":"36d1c986f33a8a550ce8cd2534107015ab2b58e1e302af2ea081bfebf2e095bc"} Jul 26 17:49:52.459379 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.459330 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-219.ec2.internal" podStartSLOduration=20.459311222 podStartE2EDuration="20.459311222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:35.423567008 +0000 UTC m=+5.822596058" watchObservedRunningTime="2026-07-26 17:49:52.459311222 +0000 UTC m=+22.858340269" Jul 26 17:49:52.459877 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.459769 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mj8vn" podStartSLOduration=2.439369074 podStartE2EDuration="21.459756458s" totalImagesPullingTime="19.020387384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:52.458727096 +0000 UTC m=+22.857756154" watchObservedRunningTime="2026-07-26 17:49:52.459756458 +0000 UTC m=+22.858785507" Jul 26 17:49:52.473680 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.473408 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g9tw7" podStartSLOduration=10.212155349 podStartE2EDuration="21.473389487s" totalImagesPullingTime="11.261234138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:52.472923973 +0000 UTC m=+22.871953019" watchObservedRunningTime="2026-07-26 17:49:52.473389487 +0000 UTC m=+22.872418534" Jul 26 17:49:52.487748 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.487611 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fqjvs" podStartSLOduration=2.525135284 podStartE2EDuration="21.487596831s" totalImagesPullingTime="18.962461547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:52.48713277 +0000 UTC m=+22.886161818" watchObservedRunningTime="2026-07-26 17:49:52.487596831 +0000 UTC m=+22.886625873" Jul 26 17:49:52.503598 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.503517 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jpchf" podStartSLOduration=2.532675901 podStartE2EDuration="21.503502853s" totalImagesPullingTime="18.970826952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:52.503425368 +0000 UTC m=+22.902454412" watchObservedRunningTime="2026-07-26 17:49:52.503502853 +0000 UTC m=+22.902531899" Jul 26 17:49:52.517309 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:52.517271 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dbvcm" podStartSLOduration=2.556439942 podStartE2EDuration="21.517255225s" totalImagesPullingTime="18.960815283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:52.516681387 +0000 UTC m=+22.915710423" watchObservedRunningTime="2026-07-26 17:49:52.517255225 +0000 UTC m=+22.916284267" Jul 26 17:49:53.202687 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.202653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:53.202687 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.202686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:53.203514 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:53.202776 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:53.203514 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:53.202875 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:53.459144 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.459063 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="bc039054fdd32fd57ed6be3599d3c0ec0917929ac3cb54369757720e07aee041" exitCode=0 Jul 26 17:49:53.459367 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.459145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"bc039054fdd32fd57ed6be3599d3c0ec0917929ac3cb54369757720e07aee041"} Jul 26 17:49:53.464201 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.464081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"5702bd903c7def98ddeeac5bed274e882d6e27385baff1ff11073863d1acb6e1"} Jul 26 17:49:53.464201 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.464161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"bd214a676a0f6d61f63751e43f8b34004e5a99d072fbe36b056adc176aa45a3e"} Jul 26 17:49:53.464344 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.464201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"e1f612f09c32712042bb5c2c5046a810cf20efad5362ed4e15f4d2d2d8dea7fe"} Jul 26 17:49:53.464344 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.464247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"5b58f6bd390f371a1d85557ffa86ee1c10e941d2dbe8a77e06cc9bd483acf1de"} Jul 26 17:49:53.464344 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.464286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"61e9e1cccc8450040e444206915d8e4a1b846221ab0c6d71f6edf2771c86e1fd"} Jul 26 17:49:53.971177 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:53.970912 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 17:49:54.078926 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.078715 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T17:49:53.970934888Z","UUID":"0b20b69a-d5f3-4714-808c-18c4dd1c72c1","Handler":null,"Name":"","Endpoint":""} Jul 26 17:49:54.080144 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.080126 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 17:49:54.080254 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.080160 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 17:49:54.202466 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.202433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:54.202676 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:54.202557 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:54.468519 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.468478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" event={"ID":"b9f61d50-93d8-4424-b167-471aa98e826b","Type":"ContainerStarted","Data":"be9f3aa33ae1d1362741c544df46b831a706aaf2c98f36cde7096fcc7e232c41"} Jul 26 17:49:54.470117 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.470090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6vm5" event={"ID":"fc558bc8-71a8-4a1a-9144-e3325e963098","Type":"ContainerStarted","Data":"d8ace80fd27f949ef9bb9be53e66567eb6072f1b01eca4e164997bfca45ade77"} Jul 26 17:49:54.484887 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:54.484777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k6vm5" podStartSLOduration=4.515490155 podStartE2EDuration="23.484758734s" totalImagesPullingTime="18.969268579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:54.484517716 +0000 UTC m=+24.883546764" watchObservedRunningTime="2026-07-26 17:49:54.484758734 +0000 UTC m=+24.883787780" Jul 26 17:49:55.202358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:55.202068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:55.202358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:55.202118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:55.202574 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:55.202416 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:55.202574 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:55.202555 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:55.476379 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:55.476288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"5a92083777f4041235f70ff5fbbdd07a2fb19f49524ddecc78272ea1bf2ed6d4"} Jul 26 17:49:55.478729 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:55.478703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" event={"ID":"b9f61d50-93d8-4424-b167-471aa98e826b","Type":"ContainerStarted","Data":"3c144a6aadcc0f67d73f265483ac6393296fff86941c495d182f20d164e8e259"} Jul 26 17:49:56.202172 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.202140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:56.202379 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:56.202254 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:56.324077 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.324038 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:56.324731 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.324704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:56.341201 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.341156 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qllq" podStartSLOduration=3.677131439 podStartE2EDuration="25.341138893s" totalImagesPullingTime="21.664007454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:55.50164392 +0000 UTC m=+25.900672965" watchObservedRunningTime="2026-07-26 17:49:56.341138893 +0000 UTC m=+26.740167937" Jul 26 17:49:56.481324 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.481240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:56.481915 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:56.481563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g9tw7" Jul 26 17:49:57.202680 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:57.202621 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:57.202874 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:57.202743 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:57.202874 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:57.202815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:57.202998 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:57.202907 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:58.203084 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:58.202845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:49:58.203687 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:58.203128 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:49:58.486525 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:58.486411 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="059439533bab0370a3d87e6985091b1ab939485f12535ea2040a0a96a4af5f93" exitCode=0 Jul 26 17:49:58.486525 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:58.486502 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"059439533bab0370a3d87e6985091b1ab939485f12535ea2040a0a96a4af5f93"} Jul 26 17:49:58.490284 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:58.490261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" event={"ID":"ee547b8b-b19d-440c-8d61-8297014e9e5a","Type":"ContainerStarted","Data":"ba73a59a7aa005a66f082708bf2dde26e5c7e840b2fba6da3e201f9c5e7cc9ed"} Jul 26 17:49:58.533117 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:58.533060 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" podStartSLOduration=8.512358883 podStartE2EDuration="27.533044435s" totalImagesPullingTime="19.020685552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:49:58.532382195 +0000 UTC m=+28.931411253" watchObservedRunningTime="2026-07-26 17:49:58.533044435 +0000 UTC m=+28.932073480" Jul 26 17:49:59.202159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.202129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:49:59.202425 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.202142 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:49:59.202425 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:59.202292 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:49:59.202425 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:49:59.202385 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:49:59.493288 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.493210 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:59.493683 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.493326 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:59.493683 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.493424 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:59.508335 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.508309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:49:59.508468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:49:59.508415 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:50:00.203625 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:00.203587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:00.203769 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:00.203675 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:00.497134 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:00.497047 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="2a259285869ef57d6235712b1ae4f7db12dedd7dc8c40eadf1e5a85cc16e1827" exitCode=0 Jul 26 17:50:00.497134 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:00.497122 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"2a259285869ef57d6235712b1ae4f7db12dedd7dc8c40eadf1e5a85cc16e1827"} Jul 26 17:50:01.202652 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:01.202612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:01.202851 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:01.202617 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:01.202851 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:01.202724 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:01.202851 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:01.202820 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:02.203170 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:02.203136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:02.203547 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:02.203237 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:02.503144 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:02.503059 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="9c7eadb29ebb068a1acacde7bcceb40bd15db540b4ec4a901bd251bd28502a59" exitCode=0 Jul 26 17:50:02.503144 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:02.503099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"9c7eadb29ebb068a1acacde7bcceb40bd15db540b4ec4a901bd251bd28502a59"} Jul 26 17:50:03.202978 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:03.202944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:03.203177 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:03.202945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:03.203177 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.203092 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:03.203177 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.203155 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:03.764390 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:03.764348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:03.764580 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.764524 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:50:03.764647 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.764603 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs podName:fa10fb0a-6825-48da-9850-e455f99ca78b nodeName:}" failed. No retries permitted until 2026-07-26 17:50:35.764581581 +0000 UTC m=+66.163610621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs") pod "network-metrics-daemon-7lx5k" (UID: "fa10fb0a-6825-48da-9850-e455f99ca78b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 17:50:03.865568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:03.865530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:03.865769 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.865699 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 17:50:03.865860 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.865785 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret podName:eb57ba8d-0081-4e01-baec-552dfcc34995 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:35.865764432 +0000 UTC m=+66.264793473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret") pod "global-pull-secret-syncer-6dpt6" (UID: "eb57ba8d-0081-4e01-baec-552dfcc34995") : object "kube-system"/"original-pull-secret" not registered Jul 26 17:50:03.966485 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:03.966444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:03.966663 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.966614 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 17:50:03.966663 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.966635 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 17:50:03.966663 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.966646 2575 projected.go:196] Error preparing data for projected volume kube-api-access-cf7gd for pod openshift-network-diagnostics/network-check-target-xq9pn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:50:03.966781 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:03.966698 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd podName:9c1b3c51-40ab-4b7c-99b5-063cfe950ad1 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:35.966683344 +0000 UTC m=+66.365712366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf7gd" (UniqueName: "kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd") pod "network-check-target-xq9pn" (UID: "9c1b3c51-40ab-4b7c-99b5-063cfe950ad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 17:50:04.202662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:04.202631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:04.202843 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:04.202736 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:05.202563 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:05.202532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:05.203008 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:05.202664 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:05.203008 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:05.202704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:05.203008 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:05.202803 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:06.202098 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:06.202058 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:06.202280 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:06.202219 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:07.202582 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:07.202547 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:07.202967 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:07.202673 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:07.202967 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:07.202710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:07.202967 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:07.202828 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:07.518223 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:07.518151 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="f7adeac74e0d8ebd33d04b1ea452d07ab8e311b514d1fb711f1ce85880ba38ef" exitCode=0 Jul 26 17:50:07.518343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:07.518215 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"f7adeac74e0d8ebd33d04b1ea452d07ab8e311b514d1fb711f1ce85880ba38ef"} Jul 26 17:50:08.202549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:08.202512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:08.202719 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:08.202623 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:08.523812 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:08.523723 2575 generic.go:356] "Generic (PLEG): container finished" podID="f41c2f6a-1953-4d57-8c13-475c8b635c5c" containerID="736c22fa9bc3978d3310a1a4b2311ed7905d613a5eab5866e57aa35ede17a137" exitCode=0 Jul 26 17:50:08.523812 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:08.523763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerDied","Data":"736c22fa9bc3978d3310a1a4b2311ed7905d613a5eab5866e57aa35ede17a137"} Jul 26 17:50:09.202328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:09.202295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:09.202328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:09.202327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:09.202522 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:09.202418 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:09.202605 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:09.202519 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:09.530619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:09.530528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" event={"ID":"f41c2f6a-1953-4d57-8c13-475c8b635c5c","Type":"ContainerStarted","Data":"bb18de18ba73758c13145783abd20fd5006923d01ec893b7b3f934d0b83ca581"} Jul 26 17:50:10.205028 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.204995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:10.205189 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:10.205119 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:10.224584 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.224536 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2lr4f" podStartSLOduration=9.703728904 podStartE2EDuration="39.224519677s" totalImagesPullingTime="29.520790773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:09.568768233 +0000 UTC m=+39.967797279" watchObservedRunningTime="2026-07-26 17:50:10.224519677 +0000 UTC m=+40.623548722" Jul 26 17:50:10.978438 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.977931 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lx5k"] Jul 26 17:50:10.978438 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.978118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:10.978438 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:10.978233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:10.979321 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.978733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq9pn"] Jul 26 17:50:10.979321 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.978857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:10.979321 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:10.978955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:10.979519 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.979500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6dpt6"] Jul 26 17:50:10.979639 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:10.979604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:10.980033 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:10.979688 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:12.202478 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:12.202443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:12.202892 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:12.202562 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:13.202416 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:13.202382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:13.202583 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:13.202428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:13.202583 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:13.202506 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xq9pn" podUID="9c1b3c51-40ab-4b7c-99b5-063cfe950ad1" Jul 26 17:50:13.202894 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:13.202622 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dpt6" podUID="eb57ba8d-0081-4e01-baec-552dfcc34995" Jul 26 17:50:14.202469 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:14.202435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:14.202629 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:14.202545 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7lx5k" podUID="fa10fb0a-6825-48da-9850-e455f99ca78b" Jul 26 17:50:14.949439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:14.949275 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-219.ec2.internal" event="NodeReady" Jul 26 17:50:14.949618 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:14.949533 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 17:50:15.015260 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.015168 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g9jrk"] Jul 26 17:50:15.033977 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.033947 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rg2p5"] Jul 26 17:50:15.034155 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.034070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.037960 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.037927 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jb7dx\"" Jul 26 17:50:15.037960 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.037927 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 17:50:15.038262 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.038242 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 17:50:15.058778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.058754 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9jrk"] Jul 26 17:50:15.058902 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.058801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rg2p5"] Jul 26 17:50:15.058902 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.058818 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k"] Jul 26 17:50:15.058902 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.058873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.061734 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.061713 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 17:50:15.061864 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.061743 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 17:50:15.061864 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.061750 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btq2h\"" Jul 26 17:50:15.061864 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.061829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 17:50:15.078884 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.078863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k"] Jul 26 17:50:15.078998 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.078949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:15.081632 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.081606 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 26 17:50:15.081783 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.081766 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d6f4h\"" Jul 26 17:50:15.147353 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7hh\" (UniqueName: \"kubernetes.io/projected/07f0a26c-bdb1-4170-aea6-884aadf1345c-kube-api-access-vb7hh\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.147353 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8709f3b-1af2-4562-946a-19504964c505-cert\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.147590 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5z4\" (UniqueName: \"kubernetes.io/projected/a8709f3b-1af2-4562-946a-19504964c505-kube-api-access-4k5z4\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.147641 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07f0a26c-bdb1-4170-aea6-884aadf1345c-metrics-tls\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.147695 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07f0a26c-bdb1-4170-aea6-884aadf1345c-config-volume\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.147743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.147699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0a26c-bdb1-4170-aea6-884aadf1345c-tmp-dir\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.202372 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.202334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:15.202542 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.202521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:15.205365 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.205344 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 17:50:15.205730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.205447 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 17:50:15.205730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.205447 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 17:50:15.205730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.205664 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nb78b\"" Jul 26 17:50:15.249047 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7hh\" (UniqueName: \"kubernetes.io/projected/07f0a26c-bdb1-4170-aea6-884aadf1345c-kube-api-access-vb7hh\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.249185 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8709f3b-1af2-4562-946a-19504964c505-cert\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.249185 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5z4\" (UniqueName: \"kubernetes.io/projected/a8709f3b-1af2-4562-946a-19504964c505-kube-api-access-4k5z4\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.249185 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07f0a26c-bdb1-4170-aea6-884aadf1345c-metrics-tls\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.249327 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07f0a26c-bdb1-4170-aea6-884aadf1345c-config-volume\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.249387 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0a26c-bdb1-4170-aea6-884aadf1345c-tmp-dir\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.249438 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9r4k\" (UID: \"adadb65e-6276-4c4f-b603-824019664192\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:15.249666 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0a26c-bdb1-4170-aea6-884aadf1345c-tmp-dir\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.249934 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.249916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/07f0a26c-bdb1-4170-aea6-884aadf1345c-config-volume\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.252877 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.252856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/07f0a26c-bdb1-4170-aea6-884aadf1345c-metrics-tls\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.252971 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.252891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8709f3b-1af2-4562-946a-19504964c505-cert\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.257413 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.257394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5z4\" (UniqueName: \"kubernetes.io/projected/a8709f3b-1af2-4562-946a-19504964c505-kube-api-access-4k5z4\") pod \"ingress-canary-rg2p5\" (UID: \"a8709f3b-1af2-4562-946a-19504964c505\") " pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.266842 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.266775 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7hh\" (UniqueName: \"kubernetes.io/projected/07f0a26c-bdb1-4170-aea6-884aadf1345c-kube-api-access-vb7hh\") pod \"dns-default-g9jrk\" (UID: \"07f0a26c-bdb1-4170-aea6-884aadf1345c\") " pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.345323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.345289 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:15.350590 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.350566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9r4k\" (UID: \"adadb65e-6276-4c4f-b603-824019664192\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:15.350687 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.350675 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 26 17:50:15.350741 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.350734 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates podName:adadb65e-6276-4c4f-b603-824019664192 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:15.850720739 +0000 UTC m=+46.249749761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" (UID: "adadb65e-6276-4c4f-b603-824019664192") : secret "prometheus-operator-admission-webhook-tls" not found Jul 26 17:50:15.367374 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.367346 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rg2p5" Jul 26 17:50:15.518147 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.518054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g9jrk"] Jul 26 17:50:15.519192 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.519175 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rg2p5"] Jul 26 17:50:15.521979 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:15.521953 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f0a26c_bdb1_4170_aea6_884aadf1345c.slice/crio-c200b8ae07abbf36bd1256ca9d95f8b66b6546c0b8863fe05582f8b9ff6f48b8 WatchSource:0}: Error finding container c200b8ae07abbf36bd1256ca9d95f8b66b6546c0b8863fe05582f8b9ff6f48b8: Status 404 returned error can't find the container with id c200b8ae07abbf36bd1256ca9d95f8b66b6546c0b8863fe05582f8b9ff6f48b8 Jul 26 17:50:15.522849 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:15.522827 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8709f3b_1af2_4562_946a_19504964c505.slice/crio-b5e37b46db29430e6162c81ca5da73bce9724a832d2077c3c7701883437b5e44 WatchSource:0}: Error finding container b5e37b46db29430e6162c81ca5da73bce9724a832d2077c3c7701883437b5e44: Status 404 returned error can't find the container with id b5e37b46db29430e6162c81ca5da73bce9724a832d2077c3c7701883437b5e44 Jul 26 17:50:15.543916 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.543883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rg2p5" event={"ID":"a8709f3b-1af2-4562-946a-19504964c505","Type":"ContainerStarted","Data":"b5e37b46db29430e6162c81ca5da73bce9724a832d2077c3c7701883437b5e44"} Jul 26 17:50:15.544811 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.544772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jrk" event={"ID":"07f0a26c-bdb1-4170-aea6-884aadf1345c","Type":"ContainerStarted","Data":"c200b8ae07abbf36bd1256ca9d95f8b66b6546c0b8863fe05582f8b9ff6f48b8"} Jul 26 17:50:15.703656 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.703533 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g5cc5"] Jul 26 17:50:15.723751 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.723717 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5cc5"] Jul 26 17:50:15.723911 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.723857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.726332 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.726311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 17:50:15.726332 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.726326 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 17:50:15.726549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.726313 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 17:50:15.726549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.726499 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 17:50:15.726740 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.726720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dk9zk\"" Jul 26 17:50:15.854264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9r4k\" (UID: \"adadb65e-6276-4c4f-b603-824019664192\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:15.854439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.854439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.854439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c284f4f-37ab-4491-9099-73320fd0c7a0-data-volume\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.854439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrxq\" (UniqueName: \"kubernetes.io/projected/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-api-access-vxrxq\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.854439 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.854388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c284f4f-37ab-4491-9099-73320fd0c7a0-crio-socket\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.857202 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.857179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adadb65e-6276-4c4f-b603-824019664192-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9r4k\" (UID: \"adadb65e-6276-4c4f-b603-824019664192\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:15.955383 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955556 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c284f4f-37ab-4491-9099-73320fd0c7a0-data-volume\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955556 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrxq\" (UniqueName: \"kubernetes.io/projected/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-api-access-vxrxq\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955556 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.955438 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 17:50:15.955556 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c284f4f-37ab-4491-9099-73320fd0c7a0-crio-socket\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955556 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.955519 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:16.45549681 +0000 UTC m=+46.854525849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : configmap "kube-rbac-proxy" not found Jul 26 17:50:15.955868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c284f4f-37ab-4491-9099-73320fd0c7a0-crio-socket\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955868 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.955747 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 17:50:15.955868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.955762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c284f4f-37ab-4491-9099-73320fd0c7a0-data-volume\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.955868 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:15.955821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:16.455780759 +0000 UTC m=+46.854809837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : secret "insights-runtime-extractor-tls" not found Jul 26 17:50:15.968781 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.968746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrxq\" (UniqueName: \"kubernetes.io/projected/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-api-access-vxrxq\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:15.986697 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:15.986656 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:16.158099 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.157770 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k"] Jul 26 17:50:16.163745 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:16.163711 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadadb65e_6276_4c4f_b603_824019664192.slice/crio-a0e37f75e3170dc381c92781dc155dcbf65f5ea4d76b81a996a9bd1c7625cb8a WatchSource:0}: Error finding container a0e37f75e3170dc381c92781dc155dcbf65f5ea4d76b81a996a9bd1c7625cb8a: Status 404 returned error can't find the container with id a0e37f75e3170dc381c92781dc155dcbf65f5ea4d76b81a996a9bd1c7625cb8a Jul 26 17:50:16.202746 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.202720 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:16.205608 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.205581 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 17:50:16.206018 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.205659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zs84s\"" Jul 26 17:50:16.461062 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.460975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:16.461062 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.461043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:16.461465 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:16.461148 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 17:50:16.461465 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:16.461200 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:17.461186624 +0000 UTC m=+47.860215647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : secret "insights-runtime-extractor-tls" not found Jul 26 17:50:16.461465 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:16.461268 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 17:50:16.461465 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:16.461331 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:17.461317248 +0000 UTC m=+47.860346274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : configmap "kube-rbac-proxy" not found Jul 26 17:50:16.549059 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:16.548987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" event={"ID":"adadb65e-6276-4c4f-b603-824019664192","Type":"ContainerStarted","Data":"a0e37f75e3170dc381c92781dc155dcbf65f5ea4d76b81a996a9bd1c7625cb8a"} Jul 26 17:50:17.468075 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:17.468040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:17.468503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:17.468137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:17.468503 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:17.468186 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 17:50:17.468503 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:17.468261 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 17:50:17.468503 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:17.468264 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:19.468244201 +0000 UTC m=+49.867273228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : configmap "kube-rbac-proxy" not found Jul 26 17:50:17.468503 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:17.468329 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:19.468311404 +0000 UTC m=+49.867340429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : secret "insights-runtime-extractor-tls" not found Jul 26 17:50:17.915027 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:17.915001 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fqjvs_223047b2-a308-42a1-b28c-5be77c548d7a/dns-node-resolver/0.log" Jul 26 17:50:18.715035 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:18.715005 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbvcm_430af05c-ecbe-4796-b18b-8c11872a3d84/node-ca/0.log" Jul 26 17:50:19.484949 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.484910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:19.485154 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.485007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:19.485154 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:19.485072 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 17:50:19.485154 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:19.485102 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 17:50:19.485154 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:19.485152 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:23.48513177 +0000 UTC m=+53.884160809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : secret "insights-runtime-extractor-tls" not found Jul 26 17:50:19.485326 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:19.485170 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:23.485161497 +0000 UTC m=+53.884190519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : configmap "kube-rbac-proxy" not found Jul 26 17:50:19.557709 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.557599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rg2p5" event={"ID":"a8709f3b-1af2-4562-946a-19504964c505","Type":"ContainerStarted","Data":"860d7de9317481f11cb89b8b9f845d37e3cd324ae7ce0f9149e4cac8e02803eb"} Jul 26 17:50:19.559427 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.559402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jrk" event={"ID":"07f0a26c-bdb1-4170-aea6-884aadf1345c","Type":"ContainerStarted","Data":"d7980046cabf27d9c36515a6484667c4269099de1ae607dd68f19a23eb2b092d"} Jul 26 17:50:19.559565 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.559452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g9jrk" event={"ID":"07f0a26c-bdb1-4170-aea6-884aadf1345c","Type":"ContainerStarted","Data":"9898631a7e839ed37dfdc8049899cc087a27f794268744cddff1ee21213e20d5"} Jul 26 17:50:19.559565 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.559483 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:19.560859 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.560821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" event={"ID":"adadb65e-6276-4c4f-b603-824019664192","Type":"ContainerStarted","Data":"505f87f2e0d45b0e177ea256483e74f5ea69daa411a3f5bed1a8d0445389358c"} Jul 26 17:50:19.561053 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.561037 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:19.565555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.565530 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" Jul 26 17:50:19.573164 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.573125 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rg2p5" podStartSLOduration=2.075400829 podStartE2EDuration="5.573112351s" totalImagesPullingTime="3.497711522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:19.572757935 +0000 UTC m=+49.971786987" watchObservedRunningTime="2026-07-26 17:50:19.573112351 +0000 UTC m=+49.972141374" Jul 26 17:50:19.589120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.589071 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g9jrk" podStartSLOduration=2.09731469 podStartE2EDuration="5.589056129s" totalImagesPullingTime="3.491741439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:19.588468197 +0000 UTC m=+49.987497257" watchObservedRunningTime="2026-07-26 17:50:19.589056129 +0000 UTC m=+49.988085173" Jul 26 17:50:19.715057 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:19.715026 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rg2p5_a8709f3b-1af2-4562-946a-19504964c505/serve-healthcheck-canary/0.log" Jul 26 17:50:20.088988 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.088939 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9r4k" podStartSLOduration=2.239137672 podStartE2EDuration="5.088921248s" totalImagesPullingTime="2.849783576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:19.605589892 +0000 UTC m=+50.004618936" watchObservedRunningTime="2026-07-26 17:50:20.088921248 +0000 UTC m=+50.487950293" Jul 26 17:50:20.089229 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.089214 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hxh2g"] Jul 26 17:50:20.100914 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.100890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hxh2g"] Jul 26 17:50:20.101040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.100975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.104210 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.104188 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 26 17:50:20.104390 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.104372 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 26 17:50:20.104450 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.104391 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 17:50:20.105437 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.105416 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 17:50:20.105437 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.105429 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 17:50:20.105608 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.105481 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5g8pj\"" Jul 26 17:50:20.190945 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.190912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.190945 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.190947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910330eb-d33c-452b-bfca-fc1730835f71-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.191130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.191047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.191130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.191078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47sk\" (UniqueName: \"kubernetes.io/projected/910330eb-d33c-452b-bfca-fc1730835f71-kube-api-access-w47sk\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.291737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.291699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.291737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.291743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47sk\" (UniqueName: \"kubernetes.io/projected/910330eb-d33c-452b-bfca-fc1730835f71-kube-api-access-w47sk\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.291956 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:20.291872 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 26 17:50:20.291956 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.291873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.291956 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:20.291940 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls podName:910330eb-d33c-452b-bfca-fc1730835f71 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:20.791918399 +0000 UTC m=+51.190947423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-hxh2g" (UID: "910330eb-d33c-452b-bfca-fc1730835f71") : secret "prometheus-operator-tls" not found Jul 26 17:50:20.292085 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.291968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910330eb-d33c-452b-bfca-fc1730835f71-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.292601 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.292575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910330eb-d33c-452b-bfca-fc1730835f71-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.295700 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.295678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.302487 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.302464 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47sk\" (UniqueName: \"kubernetes.io/projected/910330eb-d33c-452b-bfca-fc1730835f71-kube-api-access-w47sk\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.795545 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.795452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:20.797886 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:20.797861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/910330eb-d33c-452b-bfca-fc1730835f71-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-hxh2g\" (UID: \"910330eb-d33c-452b-bfca-fc1730835f71\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:21.013705 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:21.013665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" Jul 26 17:50:21.164589 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:21.164558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-hxh2g"] Jul 26 17:50:21.168095 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:21.168068 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910330eb_d33c_452b_bfca_fc1730835f71.slice/crio-a210bb674748a7d34a8bcccf6c8812d28e21010e8699d7c62036cb3faab5633c WatchSource:0}: Error finding container a210bb674748a7d34a8bcccf6c8812d28e21010e8699d7c62036cb3faab5633c: Status 404 returned error can't find the container with id a210bb674748a7d34a8bcccf6c8812d28e21010e8699d7c62036cb3faab5633c Jul 26 17:50:21.566624 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:21.566593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" event={"ID":"910330eb-d33c-452b-bfca-fc1730835f71","Type":"ContainerStarted","Data":"a210bb674748a7d34a8bcccf6c8812d28e21010e8699d7c62036cb3faab5633c"} Jul 26 17:50:23.515142 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:23.514919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:23.515142 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:23.515130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:23.515549 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:23.515070 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 17:50:23.515549 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:23.515213 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:31.51519756 +0000 UTC m=+61.914226583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : secret "insights-runtime-extractor-tls" not found Jul 26 17:50:23.515549 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:23.515227 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 17:50:23.515549 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:23.515266 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm podName:4c284f4f-37ab-4491-9099-73320fd0c7a0 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:31.515256392 +0000 UTC m=+61.914285414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5cc5" (UID: "4c284f4f-37ab-4491-9099-73320fd0c7a0") : configmap "kube-rbac-proxy" not found Jul 26 17:50:23.574156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:23.574104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" event={"ID":"910330eb-d33c-452b-bfca-fc1730835f71","Type":"ContainerStarted","Data":"af553311fe2267e7f52c0acb768286459b9781049dd0879b7542ca65eef70e53"} Jul 26 17:50:23.574321 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:23.574183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" event={"ID":"910330eb-d33c-452b-bfca-fc1730835f71","Type":"ContainerStarted","Data":"79b33cb441719a97b212c687556f2056b4159e1ac48c305af18440878326e2cd"} Jul 26 17:50:23.591075 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:23.591028 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-hxh2g" podStartSLOduration=1.6502277520000002 podStartE2EDuration="3.591012661s" totalImagesPullingTime="1.940784909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:23.590345268 +0000 UTC m=+53.989374312" watchObservedRunningTime="2026-07-26 17:50:23.591012661 +0000 UTC m=+53.990041705" Jul 26 17:50:25.453518 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.453480 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8"] Jul 26 17:50:25.455801 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.455777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.458675 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.458655 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jkhtk\"" Jul 26 17:50:25.458997 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.458978 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 26 17:50:25.459416 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.459399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 26 17:50:25.482440 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.482404 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8"] Jul 26 17:50:25.509565 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.509501 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2"] Jul 26 17:50:25.511551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.511535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.514977 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.514955 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 26 17:50:25.514977 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.514963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jqz6c\"" Jul 26 17:50:25.515167 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.515057 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 26 17:50:25.515258 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.515243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 26 17:50:25.528427 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.528402 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rjqfq"] Jul 26 17:50:25.530209 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.530303 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.530303 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.530408 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.530408 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkcm2\" (UniqueName: \"kubernetes.io/projected/21ab5303-328c-4cde-97fc-8501d82c0a41-kube-api-access-wkcm2\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.530408 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.530554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6r88\" (UniqueName: \"kubernetes.io/projected/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-api-access-m6r88\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.530554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21ab5303-328c-4cde-97fc-8501d82c0a41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.530554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.530554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.530554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.530497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.533375 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.533352 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 17:50:25.533808 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.533769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fs6k7\"" Jul 26 17:50:25.533903 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.533769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 17:50:25.533903 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.533832 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 17:50:25.535391 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.535370 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2"] Jul 26 17:50:25.631433 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-tls\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6r88\" (UniqueName: \"kubernetes.io/projected/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-api-access-m6r88\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21ab5303-328c-4cde-97fc-8501d82c0a41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-textfile\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkc7\" (UniqueName: \"kubernetes.io/projected/4ce6769a-307c-419e-aa93-95148a37983a-kube-api-access-btkc7\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.631662 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-root\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-wtmp\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.631983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkcm2\" (UniqueName: \"kubernetes.io/projected/21ab5303-328c-4cde-97fc-8501d82c0a41-kube-api-access-wkcm2\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-metrics-client-ca\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-sys\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.632715 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632715 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.632715 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21ab5303-328c-4cde-97fc-8501d82c0a41-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.632715 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.632548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.635476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.635451 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.635616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.635524 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.635616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.635555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21ab5303-328c-4cde-97fc-8501d82c0a41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.635616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.635604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.640545 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.640519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkcm2\" (UniqueName: \"kubernetes.io/projected/21ab5303-328c-4cde-97fc-8501d82c0a41-kube-api-access-wkcm2\") pod \"openshift-state-metrics-b8f49f5c6-5f9l8\" (UID: \"21ab5303-328c-4cde-97fc-8501d82c0a41\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.641628 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.641609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6r88\" (UniqueName: \"kubernetes.io/projected/6aaec6c9-157f-46e2-88b2-9e496bf29cb3-kube-api-access-m6r88\") pod \"kube-state-metrics-56d9f944b4-66zt2\" (UID: \"6aaec6c9-157f-46e2-88b2-9e496bf29cb3\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.733561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-wtmp\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-metrics-client-ca\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-sys\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-tls\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-wtmp\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-textfile\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-sys\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkc7\" (UniqueName: \"kubernetes.io/projected/4ce6769a-307c-419e-aa93-95148a37983a-kube-api-access-btkc7\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.733846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.734297 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-root\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.734297 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.733968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce6769a-307c-419e-aa93-95148a37983a-root\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.734297 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.734102 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-textfile\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.734297 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.734202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-metrics-client-ca\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.734476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.734457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.736114 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.736081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.736384 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.736365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce6769a-307c-419e-aa93-95148a37983a-node-exporter-tls\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.742080 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.742051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkc7\" (UniqueName: \"kubernetes.io/projected/4ce6769a-307c-419e-aa93-95148a37983a-kube-api-access-btkc7\") pod \"node-exporter-rjqfq\" (UID: \"4ce6769a-307c-419e-aa93-95148a37983a\") " pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.764784 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.764756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" Jul 26 17:50:25.820225 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.820190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" Jul 26 17:50:25.839130 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.838954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rjqfq" Jul 26 17:50:25.849897 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:25.849872 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce6769a_307c_419e_aa93_95148a37983a.slice/crio-b85450912f8738ac2553d68df3d607cad2bf0c6132cdc442a8d8006ed969dc87 WatchSource:0}: Error finding container b85450912f8738ac2553d68df3d607cad2bf0c6132cdc442a8d8006ed969dc87: Status 404 returned error can't find the container with id b85450912f8738ac2553d68df3d607cad2bf0c6132cdc442a8d8006ed969dc87 Jul 26 17:50:25.898580 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.898530 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8"] Jul 26 17:50:25.903099 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:25.903069 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ab5303_328c_4cde_97fc_8501d82c0a41.slice/crio-ed1f3bc7f24933201770b9aee45cf45b08984c8c18d30c301b03a68279e384a9 WatchSource:0}: Error finding container ed1f3bc7f24933201770b9aee45cf45b08984c8c18d30c301b03a68279e384a9: Status 404 returned error can't find the container with id ed1f3bc7f24933201770b9aee45cf45b08984c8c18d30c301b03a68279e384a9 Jul 26 17:50:25.958894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:25.958871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2"] Jul 26 17:50:25.961679 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:25.961654 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaec6c9_157f_46e2_88b2_9e496bf29cb3.slice/crio-1f0f59b613e7aa694c6b976ce3e8c8f71dd2682a52a21bbdf51f8c874696d29a WatchSource:0}: Error finding container 1f0f59b613e7aa694c6b976ce3e8c8f71dd2682a52a21bbdf51f8c874696d29a: Status 404 returned error can't find the container with id 1f0f59b613e7aa694c6b976ce3e8c8f71dd2682a52a21bbdf51f8c874696d29a Jul 26 17:50:26.474356 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.474323 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:50:26.477247 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.477211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.480146 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.480117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 26 17:50:26.480551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.480529 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 26 17:50:26.482274 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.482240 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b4pnc\"" Jul 26 17:50:26.483990 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.483165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 26 17:50:26.483990 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.483653 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 26 17:50:26.483990 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.483907 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 26 17:50:26.484190 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.484151 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 26 17:50:26.484657 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.484245 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 26 17:50:26.484747 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.484663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 26 17:50:26.486410 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.485355 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 26 17:50:26.500720 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.500665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:50:26.540813 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.540647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.540813 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.540699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdh5\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.540813 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.540730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.540857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.540950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541116 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541426 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541426 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541426 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.541426 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.541292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.587458 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.587418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" event={"ID":"6aaec6c9-157f-46e2-88b2-9e496bf29cb3","Type":"ContainerStarted","Data":"1f0f59b613e7aa694c6b976ce3e8c8f71dd2682a52a21bbdf51f8c874696d29a"} Jul 26 17:50:26.588970 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.588939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rjqfq" event={"ID":"4ce6769a-307c-419e-aa93-95148a37983a","Type":"ContainerStarted","Data":"b85450912f8738ac2553d68df3d607cad2bf0c6132cdc442a8d8006ed969dc87"} Jul 26 17:50:26.593142 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.593111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" event={"ID":"21ab5303-328c-4cde-97fc-8501d82c0a41","Type":"ContainerStarted","Data":"a2a52734f3619b5ad350b2705eeffab4ab4d34056c46daaf4a712255f817b5fc"} Jul 26 17:50:26.593291 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.593175 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" event={"ID":"21ab5303-328c-4cde-97fc-8501d82c0a41","Type":"ContainerStarted","Data":"fd11533ffdd9d2a4bb94df3ef2566c600cf03bc7961ec7d203803bb85d0c6441"} Jul 26 17:50:26.593291 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.593205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" event={"ID":"21ab5303-328c-4cde-97fc-8501d82c0a41","Type":"ContainerStarted","Data":"ed1f3bc7f24933201770b9aee45cf45b08984c8c18d30c301b03a68279e384a9"} Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdh5\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.643429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.642967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.646378 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.646263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.647061 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.647032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.647616 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:26.647591 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle podName:d5c5e625-a95e-46c0-9798-db6a2592dce4 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:27.147567482 +0000 UTC m=+57.546596528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4") : configmap references non-existent config key: ca-bundle.crt Jul 26 17:50:26.655216 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.655131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.655216 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.655139 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.655942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656853 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656671 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656853 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdh5\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.656971 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.656887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:26.657137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:26.657115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:27.247570 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.247529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:27.248464 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.248439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:27.401607 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.401564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:50:27.493873 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.493124 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb"] Jul 26 17:50:27.495475 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.495454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.498429 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.498408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.498648 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lh8f9\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.498683 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.498914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ft24aaiadpf6c\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.499112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.499238 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 26 17:50:27.499894 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.499308 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 26 17:50:27.509001 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.508685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb"] Jul 26 17:50:27.550305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-grpc-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/012e7260-d0d1-403f-87c8-a0e3570e718e-metrics-client-ca\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.550681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.550435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgd8\" (UniqueName: \"kubernetes.io/projected/012e7260-d0d1-403f-87c8-a0e3570e718e-kube-api-access-htgd8\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.587154 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.587128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:50:27.590201 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:27.590148 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5e625_a95e_46c0_9798_db6a2592dce4.slice/crio-5c530803a448a200b130596d994ade80916bd05055155ae2e1ac53bc116374b4 WatchSource:0}: Error finding container 5c530803a448a200b130596d994ade80916bd05055155ae2e1ac53bc116374b4: Status 404 returned error can't find the container with id 5c530803a448a200b130596d994ade80916bd05055155ae2e1ac53bc116374b4 Jul 26 17:50:27.607046 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.607009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" event={"ID":"21ab5303-328c-4cde-97fc-8501d82c0a41","Type":"ContainerStarted","Data":"bb6f5ed0ffb13659e73aee52634377d46a3f55e35c38b67ec08409a4141da880"} Jul 26 17:50:27.614826 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.613415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" event={"ID":"6aaec6c9-157f-46e2-88b2-9e496bf29cb3","Type":"ContainerStarted","Data":"cfa045374dc8b41b6bb1060080ae6a96e70c8804de1f91d57bb0c38d1d9b1a24"} Jul 26 17:50:27.614826 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.613529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" event={"ID":"6aaec6c9-157f-46e2-88b2-9e496bf29cb3","Type":"ContainerStarted","Data":"b910824fa0143669362d2fdca2f5613cf8b762b4d45e96dc07c98ed035c49a83"} Jul 26 17:50:27.616342 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.615505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"5c530803a448a200b130596d994ade80916bd05055155ae2e1ac53bc116374b4"} Jul 26 17:50:27.621231 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.621205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rjqfq" event={"ID":"4ce6769a-307c-419e-aa93-95148a37983a","Type":"ContainerStarted","Data":"4164e4e528d50e6cfa0021dcfc42c1734ffc8d78c0c0c524a0cf6432274b8b56"} Jul 26 17:50:27.630513 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.630460 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5f9l8" podStartSLOduration=1.225615355 podStartE2EDuration="2.630444692s" totalImagesPullingTime="1.404829337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:27.629719703 +0000 UTC m=+58.028748740" watchObservedRunningTime="2026-07-26 17:50:27.630444692 +0000 UTC m=+58.029473736" Jul 26 17:50:27.651502 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.651462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-grpc-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.651780 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.651759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.652519 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.652034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/012e7260-d0d1-403f-87c8-a0e3570e718e-metrics-client-ca\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653022 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.652971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/012e7260-d0d1-403f-87c8-a0e3570e718e-metrics-client-ca\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653122 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.652752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653122 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.653092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653230 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.653120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653230 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.653157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.653337 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.653249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgd8\" (UniqueName: \"kubernetes.io/projected/012e7260-d0d1-403f-87c8-a0e3570e718e-kube-api-access-htgd8\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.655263 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.655239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-grpc-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.655442 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.655395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.656012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.655921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.656464 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.656424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.657051 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.656674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-tls\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.657752 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.657333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/012e7260-d0d1-403f-87c8-a0e3570e718e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.662703 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.662678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgd8\" (UniqueName: \"kubernetes.io/projected/012e7260-d0d1-403f-87c8-a0e3570e718e-kube-api-access-htgd8\") pod \"thanos-querier-5fdcffdbfd-lnbmb\" (UID: \"012e7260-d0d1-403f-87c8-a0e3570e718e\") " pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.806919 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.806819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:27.943096 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:27.943071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb"] Jul 26 17:50:27.945372 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:27.945342 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012e7260_d0d1_403f_87c8_a0e3570e718e.slice/crio-85f3ff9f89284afa88c0b33a66175742a17ecbd19377f2224dd2028928b596ef WatchSource:0}: Error finding container 85f3ff9f89284afa88c0b33a66175742a17ecbd19377f2224dd2028928b596ef: Status 404 returned error can't find the container with id 85f3ff9f89284afa88c0b33a66175742a17ecbd19377f2224dd2028928b596ef Jul 26 17:50:28.629365 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:28.629306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" event={"ID":"6aaec6c9-157f-46e2-88b2-9e496bf29cb3","Type":"ContainerStarted","Data":"f72bcf992cd2e90c30c61475a8a69390adb438a1abe32b0703b6910944333541"} Jul 26 17:50:28.633862 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:28.632362 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ce6769a-307c-419e-aa93-95148a37983a" containerID="4164e4e528d50e6cfa0021dcfc42c1734ffc8d78c0c0c524a0cf6432274b8b56" exitCode=0 Jul 26 17:50:28.633862 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:28.632455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rjqfq" event={"ID":"4ce6769a-307c-419e-aa93-95148a37983a","Type":"ContainerDied","Data":"4164e4e528d50e6cfa0021dcfc42c1734ffc8d78c0c0c524a0cf6432274b8b56"} Jul 26 17:50:28.635192 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:28.635162 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"85f3ff9f89284afa88c0b33a66175742a17ecbd19377f2224dd2028928b596ef"} Jul 26 17:50:28.649154 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:28.649103 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-66zt2" podStartSLOduration=2.189404993 podStartE2EDuration="3.649085113s" totalImagesPullingTime="1.45968012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:28.648117682 +0000 UTC m=+59.047146728" watchObservedRunningTime="2026-07-26 17:50:28.649085113 +0000 UTC m=+59.048114186" Jul 26 17:50:29.566263 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.566231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g9jrk" Jul 26 17:50:29.640459 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.640424 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad" exitCode=0 Jul 26 17:50:29.640938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.640511 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad"} Jul 26 17:50:29.643162 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.643133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rjqfq" event={"ID":"4ce6769a-307c-419e-aa93-95148a37983a","Type":"ContainerStarted","Data":"14bb82945bceaf66f55e4736392c4a3a8b3a58085a8b98eaf086cd4cb50b48bd"} Jul 26 17:50:29.643310 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.643193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rjqfq" event={"ID":"4ce6769a-307c-419e-aa93-95148a37983a","Type":"ContainerStarted","Data":"73feb323047f88f76de8d7331df65185e53420901f991f63c91fffebcddb3ca0"} Jul 26 17:50:29.800704 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.800642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rjqfq" podStartSLOduration=3.130967916 podStartE2EDuration="4.800623297s" totalImagesPullingTime="1.570446692s" totalInitContainerRuntime="99.208689ms" isStatefulPod=true podCreationTimestamp="2026-07-26 17:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:29.690371458 +0000 UTC m=+60.089400504" watchObservedRunningTime="2026-07-26 17:50:29.800623297 +0000 UTC m=+60.199652342" Jul 26 17:50:29.801301 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.801280 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c755fcd9f-j4pdq"] Jul 26 17:50:29.806106 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.806077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.808849 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.808822 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 26 17:50:29.808849 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.808842 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rw8cf\"" Jul 26 17:50:29.809309 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.809094 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 26 17:50:29.809309 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.809196 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1uvugmm0gdrbu\"" Jul 26 17:50:29.809469 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.809377 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 26 17:50:29.809580 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.809559 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 26 17:50:29.816010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.815990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c755fcd9f-j4pdq"] Jul 26 17:50:29.873358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-metrics-server-audit-profiles\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873418 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-client-certs\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-tls\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873652 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873652 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxlj\" (UniqueName: \"kubernetes.io/projected/8547f091-de47-4e0d-a516-f058c820a624-kube-api-access-txxlj\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873735 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8547f091-de47-4e0d-a516-f058c820a624-audit-log\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.873805 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.873750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-client-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975222 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-metrics-server-audit-profiles\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975222 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-client-certs\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975396 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-tls\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975396 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975396 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxlj\" (UniqueName: \"kubernetes.io/projected/8547f091-de47-4e0d-a516-f058c820a624-kube-api-access-txxlj\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975396 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8547f091-de47-4e0d-a516-f058c820a624-audit-log\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975612 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-client-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.975743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.975726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8547f091-de47-4e0d-a516-f058c820a624-audit-log\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.977926 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.977909 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 26 17:50:29.978640 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.978626 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 26 17:50:29.978725 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.978708 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 26 17:50:29.978774 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.978723 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 26 17:50:29.978774 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.978711 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1uvugmm0gdrbu\"" Jul 26 17:50:29.984514 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.984487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxlj\" (UniqueName: \"kubernetes.io/projected/8547f091-de47-4e0d-a516-f058c820a624-kube-api-access-txxlj\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.986203 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.986149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.986436 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.986419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8547f091-de47-4e0d-a516-f058c820a624-metrics-server-audit-profiles\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.989076 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.989047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-tls\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.989183 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.989055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-client-ca-bundle\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:29.989255 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:29.989204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8547f091-de47-4e0d-a516-f058c820a624-secret-metrics-server-client-certs\") pod \"metrics-server-6c755fcd9f-j4pdq\" (UID: \"8547f091-de47-4e0d-a516-f058c820a624\") " pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:30.119314 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.119285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rw8cf\"" Jul 26 17:50:30.127468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.127403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:30.213179 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.213143 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7"] Jul 26 17:50:30.236176 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.236146 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7"] Jul 26 17:50:30.236305 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.236268 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.241330 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.241301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 26 17:50:30.241663 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.241563 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d42tx\"" Jul 26 17:50:30.277551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.277443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.277551 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.277499 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6z7s\" (UniqueName: \"kubernetes.io/projected/05cc1b09-4aca-4e83-9a7a-de0063b9b204-kube-api-access-l6z7s\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.324302 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.324276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c755fcd9f-j4pdq"] Jul 26 17:50:30.327459 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:30.327426 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8547f091_de47_4e0d_a516_f058c820a624.slice/crio-26afe738848c306864a61493c0d2dde9a56681c8ef5183c6dd0f9912bc64027e WatchSource:0}: Error finding container 26afe738848c306864a61493c0d2dde9a56681c8ef5183c6dd0f9912bc64027e: Status 404 returned error can't find the container with id 26afe738848c306864a61493c0d2dde9a56681c8ef5183c6dd0f9912bc64027e Jul 26 17:50:30.378737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.378672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.378737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.378733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6z7s\" (UniqueName: \"kubernetes.io/projected/05cc1b09-4aca-4e83-9a7a-de0063b9b204-kube-api-access-l6z7s\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.378900 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:30.378856 2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 26 17:50:30.378944 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:50:30.378937 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert podName:05cc1b09-4aca-4e83-9a7a-de0063b9b204 nodeName:}" failed. No retries permitted until 2026-07-26 17:50:30.878915935 +0000 UTC m=+61.277944966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-h6vd7" (UID: "05cc1b09-4aca-4e83-9a7a-de0063b9b204") : secret "monitoring-plugin-cert" not found Jul 26 17:50:30.388689 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.388660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6z7s\" (UniqueName: \"kubernetes.io/projected/05cc1b09-4aca-4e83-9a7a-de0063b9b204-kube-api-access-l6z7s\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.648046 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.647963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" event={"ID":"8547f091-de47-4e0d-a516-f058c820a624","Type":"ContainerStarted","Data":"26afe738848c306864a61493c0d2dde9a56681c8ef5183c6dd0f9912bc64027e"} Jul 26 17:50:30.649882 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.649847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"8f18a5937d83b394c1c033c77def8700d6237c6ec1fbc1bd99fc398763c598df"} Jul 26 17:50:30.649981 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.649914 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"7e739fbaa1765583120f1888d14deeb1aeac742d5b6fe0ca233f9d5681d62511"} Jul 26 17:50:30.649981 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.649943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"90c29f490c9356e79733ed6b4496ddce7fc53b4379d865fc6cffedb0af8de2e3"} Jul 26 17:50:30.884427 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.884394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:30.887806 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:30.887759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05cc1b09-4aca-4e83-9a7a-de0063b9b204-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-h6vd7\" (UID: \"05cc1b09-4aca-4e83-9a7a-de0063b9b204\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:31.167315 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.167279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:31.513736 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.513661 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qwpk" Jul 26 17:50:31.590401 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.590356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:31.590581 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.590444 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:31.591065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.591044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c284f4f-37ab-4491-9099-73320fd0c7a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:31.592996 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.592971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c284f4f-37ab-4491-9099-73320fd0c7a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5cc5\" (UID: \"4c284f4f-37ab-4491-9099-73320fd0c7a0\") " pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:31.635758 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.635722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dk9zk\"" Jul 26 17:50:31.645547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.644861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5cc5" Jul 26 17:50:31.657737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.657588 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:50:31.670779 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.670749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.673988 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.673964 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lv6rh\"" Jul 26 17:50:31.674131 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674019 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 26 17:50:31.674131 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674067 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 26 17:50:31.674131 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674075 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 26 17:50:31.674131 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674089 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 26 17:50:31.674404 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674388 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 26 17:50:31.674471 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674388 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 26 17:50:31.674471 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674437 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 26 17:50:31.674573 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674484 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 26 17:50:31.674648 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dq40j91i62t9g\"" Jul 26 17:50:31.674712 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674693 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 26 17:50:31.674763 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 26 17:50:31.674840 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.674804 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 26 17:50:31.675172 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.675153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 26 17:50:31.675297 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.675279 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 26 17:50:31.682722 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.682689 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 26 17:50:31.684312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.684285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:50:31.792713 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.792620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.792885 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.792829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.792885 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.792878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793029 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.792904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793029 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.792929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793029 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793153 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793199 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793242 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793286 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793330 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793330 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793410 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793410 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793410 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793523 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793523 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793523 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnsm\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793660 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.793660 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.793589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895123 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895123 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895371 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895431 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895431 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895431 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnsm\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.895899 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.895811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.896236 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.896203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.897637 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.897605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.898248 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.898210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.898862 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.898837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.899092 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.899067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.899392 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.899370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.899966 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.899942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.900173 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.900131 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.901391 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.901179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.901592 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.901570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902459 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.902903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.903334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.905156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.904328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.906732 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.906638 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.909293 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.909274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnsm\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm\") pod \"prometheus-k8s-0\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:31.984809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:31.984751 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:32.377993 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.377773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7"] Jul 26 17:50:32.387617 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:32.387592 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cc1b09_4aca_4e83_9a7a_de0063b9b204.slice/crio-e22744b98e30ef50ca5e2e80ae87abdf6cabf94c2931793988ec39290676cfe4 WatchSource:0}: Error finding container e22744b98e30ef50ca5e2e80ae87abdf6cabf94c2931793988ec39290676cfe4: Status 404 returned error can't find the container with id e22744b98e30ef50ca5e2e80ae87abdf6cabf94c2931793988ec39290676cfe4 Jul 26 17:50:32.398856 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.398832 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5cc5"] Jul 26 17:50:32.407339 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:32.406672 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c284f4f_37ab_4491_9099_73320fd0c7a0.slice/crio-366a2f67cd8bc9193b9a02c9022219ac12bd9d7cd024ff7ddd1a51ebf1aa876f WatchSource:0}: Error finding container 366a2f67cd8bc9193b9a02c9022219ac12bd9d7cd024ff7ddd1a51ebf1aa876f: Status 404 returned error can't find the container with id 366a2f67cd8bc9193b9a02c9022219ac12bd9d7cd024ff7ddd1a51ebf1aa876f Jul 26 17:50:32.639120 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.639093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:50:32.642305 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:32.642276 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24860687_19e8_4439_946f_572ff9db1fab.slice/crio-964178da67423b34e00baa08fc04ad6a8a8120ec1c26123f451339c3e568893d WatchSource:0}: Error finding container 964178da67423b34e00baa08fc04ad6a8a8120ec1c26123f451339c3e568893d: Status 404 returned error can't find the container with id 964178da67423b34e00baa08fc04ad6a8a8120ec1c26123f451339c3e568893d Jul 26 17:50:32.662197 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.662151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" event={"ID":"8547f091-de47-4e0d-a516-f058c820a624","Type":"ContainerStarted","Data":"c31b2df4d4570eccdf1cf7a5bbf818c5eba02db374e84c9dbbcf9ce25c659853"} Jul 26 17:50:32.663982 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.663955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5cc5" event={"ID":"4c284f4f-37ab-4491-9099-73320fd0c7a0","Type":"ContainerStarted","Data":"366a2f67cd8bc9193b9a02c9022219ac12bd9d7cd024ff7ddd1a51ebf1aa876f"} Jul 26 17:50:32.671244 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.671214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc"} Jul 26 17:50:32.671981 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.671873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1"} Jul 26 17:50:32.672097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.672001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52"} Jul 26 17:50:32.672097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.672045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946"} Jul 26 17:50:32.672097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.672085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d"} Jul 26 17:50:32.672572 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.672549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" event={"ID":"05cc1b09-4aca-4e83-9a7a-de0063b9b204","Type":"ContainerStarted","Data":"e22744b98e30ef50ca5e2e80ae87abdf6cabf94c2931793988ec39290676cfe4"} Jul 26 17:50:32.675763 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.675734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"17cc5f145c93b191d48c338bc4c5966478e264df36fe4e8c5109f3ca6b57e4e2"} Jul 26 17:50:32.675892 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.675814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"edd5c927a963fb90e7a65eb575dd7ff8f6eb0d48b8f075cf191975fda0d650fa"} Jul 26 17:50:32.675892 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.675861 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" event={"ID":"012e7260-d0d1-403f-87c8-a0e3570e718e","Type":"ContainerStarted","Data":"3330d4e3900c51dee5719ac251e6f3a6eddf864f3e3786e0dfe287a389824195"} Jul 26 17:50:32.676019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.675928 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:32.676918 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.676856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"964178da67423b34e00baa08fc04ad6a8a8120ec1c26123f451339c3e568893d"} Jul 26 17:50:32.759170 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.759118 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" podStartSLOduration=1.873305604 podStartE2EDuration="3.75910117s" totalImagesPullingTime="1.885795566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:32.746821566 +0000 UTC m=+63.145850610" watchObservedRunningTime="2026-07-26 17:50:32.75910117 +0000 UTC m=+63.158130209" Jul 26 17:50:32.824385 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:32.821205 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" podStartSLOduration=1.8871357560000002 podStartE2EDuration="5.821180944s" totalImagesPullingTime="3.934045188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:32.8204256 +0000 UTC m=+63.219454642" watchObservedRunningTime="2026-07-26 17:50:32.821180944 +0000 UTC m=+63.220209971" Jul 26 17:50:33.684815 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:33.684434 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" exitCode=0 Jul 26 17:50:33.684815 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:33.684561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} Jul 26 17:50:33.689946 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:33.689827 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerStarted","Data":"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9"} Jul 26 17:50:33.740330 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:33.740278 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.078418152 podStartE2EDuration="7.740261169s" totalImagesPullingTime="3.644254008s" totalInitContainerRuntime="17.589009ms" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:33.737867036 +0000 UTC m=+64.136896101" watchObservedRunningTime="2026-07-26 17:50:33.740261169 +0000 UTC m=+64.139290214" Jul 26 17:50:34.697036 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:34.696997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" event={"ID":"05cc1b09-4aca-4e83-9a7a-de0063b9b204","Type":"ContainerStarted","Data":"ecb630d0c1e43e3ac59613c186b6c89d66cd2fe7ec3ad9e733fa9bfb5e909f66"} Jul 26 17:50:34.697474 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:34.697063 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:34.703012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:34.702951 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" Jul 26 17:50:34.715455 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:34.715403 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-h6vd7" podStartSLOduration=3.262461043 podStartE2EDuration="4.715381922s" totalImagesPullingTime="1.452920879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:34.713781524 +0000 UTC m=+65.112810590" watchObservedRunningTime="2026-07-26 17:50:34.715381922 +0000 UTC m=+65.114410968" Jul 26 17:50:35.836271 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.836227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:35.839339 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.839317 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 17:50:35.848826 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.848780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa10fb0a-6825-48da-9850-e455f99ca78b-metrics-certs\") pod \"network-metrics-daemon-7lx5k\" (UID: \"fa10fb0a-6825-48da-9850-e455f99ca78b\") " pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:35.937644 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.937607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:35.940539 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.940517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 17:50:35.951472 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:35.951433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb57ba8d-0081-4e01-baec-552dfcc34995-original-pull-secret\") pod \"global-pull-secret-syncer-6dpt6\" (UID: \"eb57ba8d-0081-4e01-baec-552dfcc34995\") " pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:36.017421 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.017387 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zs84s\"" Jul 26 17:50:36.025516 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.025485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7lx5k" Jul 26 17:50:36.038399 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.038367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:36.041105 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.041082 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 17:50:36.051730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.051708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 17:50:36.062312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.062290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7gd\" (UniqueName: \"kubernetes.io/projected/9c1b3c51-40ab-4b7c-99b5-063cfe950ad1-kube-api-access-cf7gd\") pod \"network-check-target-xq9pn\" (UID: \"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1\") " pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:36.214050 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.214013 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nb78b\"" Jul 26 17:50:36.215970 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.215948 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dpt6" Jul 26 17:50:36.221844 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:36.221813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:38.698250 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:38.698224 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fdcffdbfd-lnbmb" Jul 26 17:50:39.253514 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.253251 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xq9pn"] Jul 26 17:50:39.260634 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:39.260598 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1b3c51_40ab_4b7c_99b5_063cfe950ad1.slice/crio-fabf1b9c88e46ea4580a9f83811c63d60545d30e46b162315abf3ef79ad793a6 WatchSource:0}: Error finding container fabf1b9c88e46ea4580a9f83811c63d60545d30e46b162315abf3ef79ad793a6: Status 404 returned error can't find the container with id fabf1b9c88e46ea4580a9f83811c63d60545d30e46b162315abf3ef79ad793a6 Jul 26 17:50:39.275687 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.275270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7lx5k"] Jul 26 17:50:39.280526 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:39.280145 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa10fb0a_6825_48da_9850_e455f99ca78b.slice/crio-1928328844db1b41f0328e6f3493aa47f84673aae630665919eb4632b849ff6a WatchSource:0}: Error finding container 1928328844db1b41f0328e6f3493aa47f84673aae630665919eb4632b849ff6a: Status 404 returned error can't find the container with id 1928328844db1b41f0328e6f3493aa47f84673aae630665919eb4632b849ff6a Jul 26 17:50:39.295009 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.294984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6dpt6"] Jul 26 17:50:39.297745 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:50:39.297709 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57ba8d_0081_4e01_baec_552dfcc34995.slice/crio-9b47c95846b899b912dd1963c085ef85407db3ba6e46bf7a62ad5075244bd7c1 WatchSource:0}: Error finding container 9b47c95846b899b912dd1963c085ef85407db3ba6e46bf7a62ad5075244bd7c1: Status 404 returned error can't find the container with id 9b47c95846b899b912dd1963c085ef85407db3ba6e46bf7a62ad5075244bd7c1 Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.729837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.729922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.729965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.730006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.730048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} Jul 26 17:50:39.731234 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.730086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerStarted","Data":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} Jul 26 17:50:39.737380 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.737310 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq9pn" event={"ID":"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1","Type":"ContainerStarted","Data":"fabf1b9c88e46ea4580a9f83811c63d60545d30e46b162315abf3ef79ad793a6"} Jul 26 17:50:39.742971 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.742899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5cc5" event={"ID":"4c284f4f-37ab-4491-9099-73320fd0c7a0","Type":"ContainerStarted","Data":"519fb789cffa6f801c9e7a639aa760ad5f71af60db61ac293a95f332e1d341f3"} Jul 26 17:50:39.746437 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.746375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6dpt6" event={"ID":"eb57ba8d-0081-4e01-baec-552dfcc34995","Type":"ContainerStarted","Data":"9b47c95846b899b912dd1963c085ef85407db3ba6e46bf7a62ad5075244bd7c1"} Jul 26 17:50:39.748561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:39.748509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx5k" event={"ID":"fa10fb0a-6825-48da-9850-e455f99ca78b","Type":"ContainerStarted","Data":"1928328844db1b41f0328e6f3493aa47f84673aae630665919eb4632b849ff6a"} Jul 26 17:50:41.769461 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:41.768588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx5k" event={"ID":"fa10fb0a-6825-48da-9850-e455f99ca78b","Type":"ContainerStarted","Data":"f7c444efc6521c883675c1dc0bffd5175f8fb1bfe53c79f5afc8a142608fe6e5"} Jul 26 17:50:41.769461 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:41.768657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7lx5k" event={"ID":"fa10fb0a-6825-48da-9850-e455f99ca78b","Type":"ContainerStarted","Data":"7a418db1284b564ef1df912cb780c3962bb11a5121082b72039c6ad2549ed69b"} Jul 26 17:50:41.788148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:41.788033 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7lx5k" podStartSLOduration=69.386056493 podStartE2EDuration="1m10.788016701s" totalImagesPullingTime="1.401960208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:41.786160356 +0000 UTC m=+72.185189413" watchObservedRunningTime="2026-07-26 17:50:41.788016701 +0000 UTC m=+72.187045744" Jul 26 17:50:41.788148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:41.788133 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.368117109 podStartE2EDuration="10.788122393s" totalImagesPullingTime="5.402774998s" totalInitContainerRuntime="17.230286ms" isStatefulPod=false podCreationTimestamp="2026-07-26 17:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:39.774828519 +0000 UTC m=+70.173857557" watchObservedRunningTime="2026-07-26 17:50:41.788122393 +0000 UTC m=+72.187151440" Jul 26 17:50:41.984908 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:41.984871 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:50:46.795222 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.795176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xq9pn" event={"ID":"9c1b3c51-40ab-4b7c-99b5-063cfe950ad1","Type":"ContainerStarted","Data":"75c8309ac3a8b55e9c5be1f628b520a1d364501a3d10b1f07182d960ec5e5ca2"} Jul 26 17:50:46.795642 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.795272 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:50:46.797469 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.797441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5cc5" event={"ID":"4c284f4f-37ab-4491-9099-73320fd0c7a0","Type":"ContainerStarted","Data":"7968d5bb9b79a91dd4f9fcbd8299f2ad34c670ee5729751a3b6a79476d9bd4bd"} Jul 26 17:50:46.799137 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.799110 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6dpt6" event={"ID":"eb57ba8d-0081-4e01-baec-552dfcc34995","Type":"ContainerStarted","Data":"c25cc52080a71f7e4a3c6cdf0f3cf4e6ae0726003ed5ebd88cebfe0305647256"} Jul 26 17:50:46.811626 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.811568 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xq9pn" podStartSLOduration=69.158938067 podStartE2EDuration="1m15.811548053s" totalImagesPullingTime="6.652609986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:46.811151487 +0000 UTC m=+77.210180532" watchObservedRunningTime="2026-07-26 17:50:46.811548053 +0000 UTC m=+77.210577102" Jul 26 17:50:46.829384 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:46.829324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6dpt6" podStartSLOduration=69.202231736 podStartE2EDuration="1m15.829309336s" totalImagesPullingTime="6.6270776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:46.829054617 +0000 UTC m=+77.228083663" watchObservedRunningTime="2026-07-26 17:50:46.829309336 +0000 UTC m=+77.228338384" Jul 26 17:50:49.815119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:49.815033 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5cc5" event={"ID":"4c284f4f-37ab-4491-9099-73320fd0c7a0","Type":"ContainerStarted","Data":"1bc875b98c15e904ca562aed0c6813e3bcea0a9897dd79c7bd1828c9c5ef795d"} Jul 26 17:50:49.834040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:49.833975 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g5cc5" podStartSLOduration=17.835986199 podStartE2EDuration="34.833954707s" totalImagesPullingTime="16.997968508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 17:50:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:50:49.83199363 +0000 UTC m=+80.231022677" watchObservedRunningTime="2026-07-26 17:50:49.833954707 +0000 UTC m=+80.232983754" Jul 26 17:50:50.128182 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:50.128148 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:50:50.128379 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:50:50.128217 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:51:10.133373 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:10.133255 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:51:10.137045 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:10.137021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c755fcd9f-j4pdq" Jul 26 17:51:17.807052 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:17.807021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xq9pn" Jul 26 17:51:31.985402 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:31.985363 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:32.004928 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:32.004904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:33.000893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:33.000866 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:45.723378 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723338 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723701 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="alertmanager" containerID="cri-o://ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d" gracePeriod=120 Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723764 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-metric" containerID="cri-o://4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc" gracePeriod=120 Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723777 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-web" containerID="cri-o://37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52" gracePeriod=120 Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723839 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="prom-label-proxy" containerID="cri-o://d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9" gracePeriod=120 Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723829 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy" containerID="cri-o://a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1" gracePeriod=120 Jul 26 17:51:45.724017 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:45.723853 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="config-reloader" containerID="cri-o://bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946" gracePeriod=120 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039449 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9" exitCode=0 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039476 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc" exitCode=0 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039486 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1" exitCode=0 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039495 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946" exitCode=0 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039503 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d" exitCode=0 Jul 26 17:51:46.039547 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9"} Jul 26 17:51:46.039876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039591 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc"} Jul 26 17:51:46.039876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1"} Jul 26 17:51:46.039876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039643 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946"} Jul 26 17:51:46.039876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.039667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d"} Jul 26 17:51:46.971189 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:46.971162 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.047716 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.047622 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerID="37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52" exitCode=0 Jul 26 17:51:47.047716 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.047658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52"} Jul 26 17:51:47.047716 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.047697 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.048016 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.047747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5c5e625-a95e-46c0-9798-db6a2592dce4","Type":"ContainerDied","Data":"5c530803a448a200b130596d994ade80916bd05055155ae2e1ac53bc116374b4"} Jul 26 17:51:47.048016 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.047825 2575 scope.go:117] "RemoveContainer" containerID="d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9" Jul 26 17:51:47.055676 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.055658 2575 scope.go:117] "RemoveContainer" containerID="4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc" Jul 26 17:51:47.062846 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.062825 2575 scope.go:117] "RemoveContainer" containerID="a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1" Jul 26 17:51:47.066021 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.065996 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066129 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066037 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066129 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066065 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066129 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066090 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066129 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066213 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpdh5\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066243 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066355 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:47.066555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config\") pod \"d5c5e625-a95e-46c0-9798-db6a2592dce4\" (UID: \"d5c5e625-a95e-46c0-9798-db6a2592dce4\") " Jul 26 17:51:47.066722 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.066696 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.067225 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.067198 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:51:47.068806 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.067849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:47.070462 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume" (OuterVolumeSpecName: "config-volume") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.070462 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070260 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.070462 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070369 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:51:47.070669 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.070729 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070710 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5" (OuterVolumeSpecName: "kube-api-access-tpdh5") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "kube-api-access-tpdh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:51:47.070808 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070720 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.070925 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070898 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.071020 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.070942 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out" (OuterVolumeSpecName: "config-out") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:51:47.072368 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.072348 2575 scope.go:117] "RemoveContainer" containerID="37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52" Jul 26 17:51:47.075620 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.075596 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.083174 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.083146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config" (OuterVolumeSpecName: "web-config") pod "d5c5e625-a95e-46c0-9798-db6a2592dce4" (UID: "d5c5e625-a95e-46c0-9798-db6a2592dce4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:47.090301 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.090283 2575 scope.go:117] "RemoveContainer" containerID="bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946" Jul 26 17:51:47.097570 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.097554 2575 scope.go:117] "RemoveContainer" containerID="ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d" Jul 26 17:51:47.104800 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.104768 2575 scope.go:117] "RemoveContainer" containerID="c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad" Jul 26 17:51:47.111613 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.111595 2575 scope.go:117] "RemoveContainer" containerID="d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9" Jul 26 17:51:47.111947 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.111921 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9\": container with ID starting with d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9 not found: ID does not exist" containerID="d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9" Jul 26 17:51:47.112019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.111956 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9"} err="failed to get container status \"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9\": rpc error: code = NotFound desc = could not find container \"d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9\": container with ID starting with d1e149c7d4c5fd1c4e4839c7a5badb2525886672c559ead21d04a853ebf3fce9 not found: ID does not exist" Jul 26 17:51:47.112019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.111991 2575 scope.go:117] "RemoveContainer" containerID="4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc" Jul 26 17:51:47.112244 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.112224 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc\": container with ID starting with 4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc not found: ID does not exist" containerID="4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc" Jul 26 17:51:47.112328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112250 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc"} err="failed to get container status \"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc\": rpc error: code = NotFound desc = could not find container \"4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc\": container with ID starting with 4500f0a07bf1d377c633ffe7145f800e0a9403a8c812bc7588499833dfbaf9dc not found: ID does not exist" Jul 26 17:51:47.112328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112267 2575 scope.go:117] "RemoveContainer" containerID="a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1" Jul 26 17:51:47.112556 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.112532 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1\": container with ID starting with a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1 not found: ID does not exist" containerID="a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1" Jul 26 17:51:47.112625 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112566 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1"} err="failed to get container status \"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1\": rpc error: code = NotFound desc = could not find container \"a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1\": container with ID starting with a382da6ae177b1ab22a71ab06df6ab85e6095ad88068a45cb4cfb1f75b6dfdc1 not found: ID does not exist" Jul 26 17:51:47.112625 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112588 2575 scope.go:117] "RemoveContainer" containerID="37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52" Jul 26 17:51:47.112878 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.112855 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52\": container with ID starting with 37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52 not found: ID does not exist" containerID="37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52" Jul 26 17:51:47.112946 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112888 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52"} err="failed to get container status \"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52\": rpc error: code = NotFound desc = could not find container \"37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52\": container with ID starting with 37ce3765b48aa02a5018222adb8545aee77532f44e7a804a939ec2e27d03bf52 not found: ID does not exist" Jul 26 17:51:47.112946 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.112904 2575 scope.go:117] "RemoveContainer" containerID="bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946" Jul 26 17:51:47.113186 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.113167 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946\": container with ID starting with bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946 not found: ID does not exist" containerID="bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946" Jul 26 17:51:47.113235 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.113192 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946"} err="failed to get container status \"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946\": rpc error: code = NotFound desc = could not find container \"bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946\": container with ID starting with bddce916b8955a29840aba1f7fcf07bfc369746b3bbef9e0ca2e5cb1ba640946 not found: ID does not exist" Jul 26 17:51:47.113235 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.113208 2575 scope.go:117] "RemoveContainer" containerID="ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d" Jul 26 17:51:47.113465 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.113435 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d\": container with ID starting with ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d not found: ID does not exist" containerID="ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d" Jul 26 17:51:47.113543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.113463 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d"} err="failed to get container status \"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d\": rpc error: code = NotFound desc = could not find container \"ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d\": container with ID starting with ab098eaf27f54e14cd4784ae428903c5964cfafe269a7e66d7ea43f66e55169d not found: ID does not exist" Jul 26 17:51:47.113543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.113482 2575 scope.go:117] "RemoveContainer" containerID="c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad" Jul 26 17:51:47.113709 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:47.113690 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad\": container with ID starting with c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad not found: ID does not exist" containerID="c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad" Jul 26 17:51:47.113813 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.113709 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad"} err="failed to get container status \"c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad\": rpc error: code = NotFound desc = could not find container \"c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad\": container with ID starting with c55974da430fe84498881b225c3cb0276dae0696f1f91a6df49fe07a3e9802ad not found: ID does not exist" Jul 26 17:51:47.167391 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167354 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdh5\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-kube-api-access-tpdh5\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167391 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167383 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-main-tls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167391 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167393 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167403 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-alertmanager-main-db\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167413 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5c5e625-a95e-46c0-9798-db6a2592dce4-tls-assets\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167422 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-web-config\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167431 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-cluster-tls-config\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167441 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167449 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-volume\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167457 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5e625-a95e-46c0-9798-db6a2592dce4-metrics-client-ca\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167467 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5c5e625-a95e-46c0-9798-db6a2592dce4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.167609 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.167476 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5c5e625-a95e-46c0-9798-db6a2592dce4-config-out\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:47.372371 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.372340 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:47.376680 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.376656 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:47.402411 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402375 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:47.402684 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402671 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="alertmanager" Jul 26 17:51:47.402730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402686 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="alertmanager" Jul 26 17:51:47.402730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402695 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-web" Jul 26 17:51:47.402730 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402701 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-web" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.402998 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="init-config-reloader" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403017 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="init-config-reloader" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403033 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-metric" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403054 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-metric" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403081 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="config-reloader" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403089 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="config-reloader" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403125 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403133 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403205 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="prom-label-proxy" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403227 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="prom-label-proxy" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403433 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403457 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-web" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403467 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="kube-rbac-proxy-metric" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403475 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="alertmanager" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403492 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="config-reloader" Jul 26 17:51:47.406809 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.403502 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" containerName="prom-label-proxy" Jul 26 17:51:47.410627 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.410599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.413597 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413570 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 26 17:51:47.413712 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413636 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-b4pnc\"" Jul 26 17:51:47.413712 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 26 17:51:47.413712 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413573 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 26 17:51:47.413712 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413667 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 26 17:51:47.413935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413575 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 26 17:51:47.413935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413818 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 26 17:51:47.413935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.413844 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 26 17:51:47.414183 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.414160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 26 17:51:47.420016 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.419998 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 26 17:51:47.420606 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.420586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:47.470318 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470318 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-out\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvtk\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-kube-api-access-qtvtk\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-volume\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.470764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.470651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-web-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.571768 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.571768 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvtk\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-kube-api-access-qtvtk\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-volume\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-web-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572006 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.571999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-out\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.572358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.573066 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572817 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.573066 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.572918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.573255 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.573122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575195 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575015 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575195 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-volume\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575195 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575195 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575195 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-web-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575400 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2109ce76-8557-4b0b-b98e-7cd3602ff608-config-out\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.575537 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.575521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.576836 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.576819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2109ce76-8557-4b0b-b98e-7cd3602ff608-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.580663 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.580643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvtk\" (UniqueName: \"kubernetes.io/projected/2109ce76-8557-4b0b-b98e-7cd3602ff608-kube-api-access-qtvtk\") pod \"alertmanager-main-0\" (UID: \"2109ce76-8557-4b0b-b98e-7cd3602ff608\") " pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.722522 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.722435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 26 17:51:47.862771 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:47.862735 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 26 17:51:47.865947 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:51:47.865914 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2109ce76_8557_4b0b_b98e_7cd3602ff608.slice/crio-5c98fc77f0981884e90e88100f0c844809077a94c634d19f8e2a795fb7e73fa0 WatchSource:0}: Error finding container 5c98fc77f0981884e90e88100f0c844809077a94c634d19f8e2a795fb7e73fa0: Status 404 returned error can't find the container with id 5c98fc77f0981884e90e88100f0c844809077a94c634d19f8e2a795fb7e73fa0 Jul 26 17:51:48.053356 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:48.053258 2575 generic.go:356] "Generic (PLEG): container finished" podID="2109ce76-8557-4b0b-b98e-7cd3602ff608" containerID="43a0f947cd42b566499b39f890e3970f02e5c5d9b2e4dcbdbe05189542ce57e9" exitCode=0 Jul 26 17:51:48.053816 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:48.053353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerDied","Data":"43a0f947cd42b566499b39f890e3970f02e5c5d9b2e4dcbdbe05189542ce57e9"} Jul 26 17:51:48.053816 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:48.053425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"5c98fc77f0981884e90e88100f0c844809077a94c634d19f8e2a795fb7e73fa0"} Jul 26 17:51:48.207676 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:48.207647 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c5e625-a95e-46c0-9798-db6a2592dce4" path="/var/lib/kubelet/pods/d5c5e625-a95e-46c0-9798-db6a2592dce4/volumes" Jul 26 17:51:49.062213 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"caf08f8fc17417a41da316b9e69503b47cb379d19e2041540850fa5f5876440e"} Jul 26 17:51:49.062631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"49a94fa9f0c4e48e5d4c3e684378f88f7bff03a69396e8d1834c8d14656cab72"} Jul 26 17:51:49.062631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"48af86beb7bf95d7422798c72401792a51b2064643f5393513cc05ab78eac2a7"} Jul 26 17:51:49.062631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"1f9b3ddd220080aba1781a39174b2ab24006b67c236d7607dab3f14388bf7dc0"} Jul 26 17:51:49.062631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"83d00ed4de7a88ddcd7f6c1c603e2864f9e532c6d18a190af5a0b65af605c5ef"} Jul 26 17:51:49.062631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.062331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2109ce76-8557-4b0b-b98e-7cd3602ff608","Type":"ContainerStarted","Data":"21422bdb64e71290ab4c7822d93fa99b86047d0ca8168ac88309bbdb243ac836"} Jul 26 17:51:49.093777 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:49.093695 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.078921953 podStartE2EDuration="2.093670588s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.748635ms" isStatefulPod=false podCreationTimestamp="2026-07-26 17:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:51:49.091143788 +0000 UTC m=+139.490172834" watchObservedRunningTime="2026-07-26 17:51:49.093670588 +0000 UTC m=+139.492699635" Jul 26 17:51:50.048166 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048441 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="prometheus" containerID="cri-o://735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" gracePeriod=600 Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048460 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-web" containerID="cri-o://4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" gracePeriod=600 Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048458 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy" containerID="cri-o://f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" gracePeriod=600 Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048492 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="config-reloader" containerID="cri-o://99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" gracePeriod=600 Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048543 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="thanos-sidecar" containerID="cri-o://3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" gracePeriod=600 Jul 26 17:51:50.048623 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.048534 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-thanos" containerID="cri-o://594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" gracePeriod=600 Jul 26 17:51:50.331019 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.330993 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:50.402263 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402417 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402276 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402417 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402298 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402417 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402319 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402417 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402417 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402386 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402424 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402457 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402484 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402631 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402587 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402633 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klnsm\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402671 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402701 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402748 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402780 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402835 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402871 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.402938 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.402909 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config\") pod \"24860687-19e8-4439-946f-572ff9db1fab\" (UID: \"24860687-19e8-4439-946f-572ff9db1fab\") " Jul 26 17:51:50.403370 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.403169 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.403370 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.403248 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.403555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.403506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.404477 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.404242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.404698 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.404678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.405419 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.405165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.406012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.405249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 17:51:50.406012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.405944 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407324 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407298 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config" (OuterVolumeSpecName: "config") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407324 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407384 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407324 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407395 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm" (OuterVolumeSpecName: "kube-api-access-klnsm") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "kube-api-access-klnsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:51:50.407681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407477 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out" (OuterVolumeSpecName: "config-out") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:51:50.407681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407590 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407681 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407854 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.407930 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.407907 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.408059 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.408031 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:51:50.408441 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.408416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.408863 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.408764 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:51:50.418882 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.418852 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config" (OuterVolumeSpecName: "web-config") pod "24860687-19e8-4439-946f-572ff9db1fab" (UID: "24860687-19e8-4439-946f-572ff9db1fab"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 17:51:50.504228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504191 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-web-config\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504223 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504239 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504254 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504269 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-config-out\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504281 2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-kube-rbac-proxy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504292 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504304 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-configmap-metrics-client-ca\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504316 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504328 2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504340 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnsm\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-kube-api-access-klnsm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504351 2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-grpc-tls\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504363 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504374 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24860687-19e8-4439-946f-572ff9db1fab-tls-assets\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504385 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504396 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504408 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24860687-19e8-4439-946f-572ff9db1fab-prometheus-k8s-db\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504420 2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-config\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:50.504468 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:50.504433 2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24860687-19e8-4439-946f-572ff9db1fab-secret-metrics-client-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:51:51.078749 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078719 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" exitCode=0 Jul 26 17:51:51.078749 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078746 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" exitCode=0 Jul 26 17:51:51.078749 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078752 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" exitCode=0 Jul 26 17:51:51.078749 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078758 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" exitCode=0 Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078764 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" exitCode=0 Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078769 2575 generic.go:356] "Generic (PLEG): container finished" podID="24860687-19e8-4439-946f-572ff9db1fab" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" exitCode=0 Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078814 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.078996 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.079025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} Jul 26 17:51:51.079097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.079063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} Jul 26 17:51:51.079480 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.079103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} Jul 26 17:51:51.079480 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.079145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24860687-19e8-4439-946f-572ff9db1fab","Type":"ContainerDied","Data":"964178da67423b34e00baa08fc04ad6a8a8120ec1c26123f451339c3e568893d"} Jul 26 17:51:51.089169 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.089150 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.096758 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.096738 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.103426 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.103407 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.107142 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.107121 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:51.111309 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.111288 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.112232 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.112213 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:51.118231 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.118210 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.125580 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.125566 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.132384 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.132363 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.132638 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.132619 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.132696 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.132652 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.132696 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.132672 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.132934 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.132918 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.132973 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.132939 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.132973 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.132953 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.133183 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.133164 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.133232 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133185 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.133232 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133197 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.133401 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.133386 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.133455 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133405 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.133455 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133417 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.133621 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.133606 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.133656 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133626 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.133656 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133639 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.133965 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.133942 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.134049 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133968 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.134049 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.133982 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.134232 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:51:51.134215 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.134345 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134239 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.134345 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134259 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.134497 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134476 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.134541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134499 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.134717 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134700 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.134817 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134719 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.134976 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134958 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.135043 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.134976 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.135232 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135215 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.135275 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135234 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.135432 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135414 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.135501 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135433 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.135668 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135639 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.135668 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135668 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.135948 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135931 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.135948 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.135948 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.136155 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136136 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.136205 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136157 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.136343 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136328 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.136381 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136345 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.136542 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136527 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.136542 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136541 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.136768 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136753 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.136768 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136767 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.137016 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.136996 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.137054 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137017 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.137240 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137214 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.137240 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137233 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.137451 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137434 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.137518 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137451 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.137669 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137651 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.137714 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137669 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.137935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137910 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.137935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.137929 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.138367 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.138197 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.138367 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.138235 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.138979 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.138952 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.138979 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.138979 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.139261 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139242 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.139340 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139270 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.139503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139483 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.139503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139502 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.139742 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139714 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.139742 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139735 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.139994 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139977 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.140051 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.139995 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.140212 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140187 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.140212 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140215 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.140458 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140441 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.140458 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140458 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.140557 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140471 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:51.140664 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140645 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.140715 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140664 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.140842 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140828 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-thanos" Jul 26 17:51:51.140888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140844 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-thanos" Jul 26 17:51:51.140888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140858 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="config-reloader" Jul 26 17:51:51.140888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140864 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="config-reloader" Jul 26 17:51:51.140888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="prometheus" Jul 26 17:51:51.140888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140889 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="prometheus" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140898 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="init-config-reloader" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140905 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="init-config-reloader" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140913 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-web" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140919 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-web" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140926 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140919 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140948 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.140931 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141029 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="thanos-sidecar" Jul 26 17:51:51.141119 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141042 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="thanos-sidecar" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141132 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141144 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="config-reloader" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141155 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="thanos-sidecar" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141167 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-web" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141176 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="prometheus" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141186 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="24860687-19e8-4439-946f-572ff9db1fab" containerName="kube-rbac-proxy-thanos" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141172 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141257 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141505 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.141617 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141521 2575 scope.go:117] "RemoveContainer" containerID="594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e" Jul 26 17:51:51.142113 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141708 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e"} err="failed to get container status \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": rpc error: code = NotFound desc = could not find container \"594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e\": container with ID starting with 594cafb9bb6fe6e9b757ce8f930b229a5fd84c1502af8dbb4a4b2d2294e3308e not found: ID does not exist" Jul 26 17:51:51.142113 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141729 2575 scope.go:117] "RemoveContainer" containerID="f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f" Jul 26 17:51:51.142113 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141968 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f"} err="failed to get container status \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": rpc error: code = NotFound desc = could not find container \"f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f\": container with ID starting with f979e35a1eaca6dd011299cfe8e73214595d56576a60329667162fdc0ee4893f not found: ID does not exist" Jul 26 17:51:51.142113 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.141984 2575 scope.go:117] "RemoveContainer" containerID="4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257" Jul 26 17:51:51.142311 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142180 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257"} err="failed to get container status \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": rpc error: code = NotFound desc = could not find container \"4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257\": container with ID starting with 4f0475d6be5055629693865afa329bcd69809ef9d68ab37fb7e69f25065b9257 not found: ID does not exist" Jul 26 17:51:51.142311 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142199 2575 scope.go:117] "RemoveContainer" containerID="3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58" Jul 26 17:51:51.142433 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142414 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58"} err="failed to get container status \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": rpc error: code = NotFound desc = could not find container \"3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58\": container with ID starting with 3f7c92940094c0b71820ce605cc3b46e6db661e6a85c136e8e0e5aaba2382c58 not found: ID does not exist" Jul 26 17:51:51.142492 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142432 2575 scope.go:117] "RemoveContainer" containerID="99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1" Jul 26 17:51:51.142616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142601 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1"} err="failed to get container status \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": rpc error: code = NotFound desc = could not find container \"99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1\": container with ID starting with 99879c98f65dd88c97aef019830dc08c671b6b121069df10b7e521e5c6e46ec1 not found: ID does not exist" Jul 26 17:51:51.142671 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142615 2575 scope.go:117] "RemoveContainer" containerID="735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185" Jul 26 17:51:51.142814 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142767 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185"} err="failed to get container status \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": rpc error: code = NotFound desc = could not find container \"735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185\": container with ID starting with 735be76e05471e950d0179202d00c5f4c979509fb9621032aeb846d5cb0db185 not found: ID does not exist" Jul 26 17:51:51.142886 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.142816 2575 scope.go:117] "RemoveContainer" containerID="6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec" Jul 26 17:51:51.143029 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.143014 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec"} err="failed to get container status \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": rpc error: code = NotFound desc = could not find container \"6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec\": container with ID starting with 6ea7c326edbf87c70a4dd5277ce176b8a925fee2c23f139fb5f2eda7471416ec not found: ID does not exist" Jul 26 17:51:51.147104 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.147083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.150472 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.150336 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.150775 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.150914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.151139 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.151239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.151459 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.151746 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 26 17:51:51.152273 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152080 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lv6rh\"" Jul 26 17:51:51.152778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152497 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 26 17:51:51.152778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152573 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dq40j91i62t9g\"" Jul 26 17:51:51.152778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152613 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 26 17:51:51.152778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 26 17:51:51.152778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152721 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 26 17:51:51.153155 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.152895 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 26 17:51:51.154312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.154294 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 26 17:51:51.157534 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.157512 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 26 17:51:51.163566 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.163548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:51.311860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.311860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-config-out\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.311860 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hk8\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-kube-api-access-s6hk8\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.311990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312184 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312465 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312616 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312710 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-web-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.312742 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.312723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413121 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413121 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-config-out\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hk8\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-kube-api-access-s6hk8\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413541 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.413893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-web-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.413956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.414013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.414231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.414323 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.414239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.416305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.416443 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.416607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.416847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.416882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.417142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417648 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.417339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3001111b-f282-4266-8c6a-84079ecf8555-config-out\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417648 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.417396 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417648 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.417466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.417893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.417816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.418065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.418042 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.418232 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.418207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3001111b-f282-4266-8c6a-84079ecf8555-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.418917 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.418887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.419344 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.419322 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.419772 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.419754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.419868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.419838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3001111b-f282-4266-8c6a-84079ecf8555-web-config\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.421543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.421523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hk8\" (UniqueName: \"kubernetes.io/projected/3001111b-f282-4266-8c6a-84079ecf8555-kube-api-access-s6hk8\") pod \"prometheus-k8s-0\" (UID: \"3001111b-f282-4266-8c6a-84079ecf8555\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.459858 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.459818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:51:51.600775 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:51.600749 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 26 17:51:51.602653 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:51:51.602629 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3001111b_f282_4266_8c6a_84079ecf8555.slice/crio-d57afd1db16d1368a6025704234925f9fd7807a8243f2c2241527908f22791f7 WatchSource:0}: Error finding container d57afd1db16d1368a6025704234925f9fd7807a8243f2c2241527908f22791f7: Status 404 returned error can't find the container with id d57afd1db16d1368a6025704234925f9fd7807a8243f2c2241527908f22791f7 Jul 26 17:51:52.084893 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:52.084853 2575 generic.go:356] "Generic (PLEG): container finished" podID="3001111b-f282-4266-8c6a-84079ecf8555" containerID="f8914b2b4dc9103a7d0f067cea2b18c76f76bd432b996867542bcc5d76e146b2" exitCode=0 Jul 26 17:51:52.085089 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:52.084952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerDied","Data":"f8914b2b4dc9103a7d0f067cea2b18c76f76bd432b996867542bcc5d76e146b2"} Jul 26 17:51:52.085089 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:52.085023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"d57afd1db16d1368a6025704234925f9fd7807a8243f2c2241527908f22791f7"} Jul 26 17:51:52.220877 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:52.220851 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24860687-19e8-4439-946f-572ff9db1fab" path="/var/lib/kubelet/pods/24860687-19e8-4439-946f-572ff9db1fab/volumes" Jul 26 17:51:53.093766 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"d2483a4e0dd5d0d89e3da999f69232a58d416935f43c26ac3d7f501726c5c79a"} Jul 26 17:51:53.094159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"407b33bcc425f6d9347262cc5619a0b3d72d5d1ff800c0e3ea170cbdc59a4000"} Jul 26 17:51:53.094159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"cd1a4387b80e4132af5760ba7a67e2740331dde4fd5af77c5b07bb9ce9d5abc1"} Jul 26 17:51:53.094159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"250d3dfc1fbf1d52ab9c2cbd79ce6a946341a9e18aa3e766213b409942348b59"} Jul 26 17:51:53.094159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093909 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"ba43697de1a7dcf17902b186225079f6568162fda89b63ed9015635259bfdf41"} Jul 26 17:51:53.094159 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.093940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3001111b-f282-4266-8c6a-84079ecf8555","Type":"ContainerStarted","Data":"521a32f29fc8ce97763dcd949570d52d8b9362c752dd113cee8659025732676c"} Jul 26 17:51:53.125278 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:53.125159 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.105412729 podStartE2EDuration="2.125138057s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.725328ms" isStatefulPod=false podCreationTimestamp="2026-07-26 17:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:51:53.123601794 +0000 UTC m=+143.522630903" watchObservedRunningTime="2026-07-26 17:51:53.125138057 +0000 UTC m=+143.524167111" Jul 26 17:51:56.460806 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:51:56.460764 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:52:51.460710 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:52:51.460662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:52:51.476007 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:52:51.475983 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:52:52.351743 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:52:52.351716 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 26 17:54:30.014872 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:30.014843 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 17:54:45.532113 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.532076 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-npx6t"] Jul 26 17:54:45.538757 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.538724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.541436 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.541417 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 26 17:54:45.541553 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.541480 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 26 17:54:45.541634 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.541599 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gznsd\"" Jul 26 17:54:45.549180 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.549156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-npx6t"] Jul 26 17:54:45.697023 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.696981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4m2s\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-kube-api-access-z4m2s\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.697210 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.697056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-bound-sa-token\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.798716 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.798616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-bound-sa-token\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.799782 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.799522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4m2s\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-kube-api-access-z4m2s\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.807601 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.807570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-bound-sa-token\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.807745 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.807727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4m2s\" (UniqueName: \"kubernetes.io/projected/d5a7ca1f-e921-4988-a2e9-b64635b29cd0-kube-api-access-z4m2s\") pod \"cert-manager-55b4d954d5-npx6t\" (UID: \"d5a7ca1f-e921-4988-a2e9-b64635b29cd0\") " pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.861558 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.861524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-npx6t" Jul 26 17:54:45.988419 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.988392 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-npx6t"] Jul 26 17:54:45.991260 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:54:45.991230 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5a7ca1f_e921_4988_a2e9_b64635b29cd0.slice/crio-98416c01e7258c2c21e66478d489ce0f82ada77054c6491f0e5b802d7fec9c05 WatchSource:0}: Error finding container 98416c01e7258c2c21e66478d489ce0f82ada77054c6491f0e5b802d7fec9c05: Status 404 returned error can't find the container with id 98416c01e7258c2c21e66478d489ce0f82ada77054c6491f0e5b802d7fec9c05 Jul 26 17:54:45.993060 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:45.993046 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 17:54:46.787649 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:46.787608 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-npx6t" event={"ID":"d5a7ca1f-e921-4988-a2e9-b64635b29cd0","Type":"ContainerStarted","Data":"98416c01e7258c2c21e66478d489ce0f82ada77054c6491f0e5b802d7fec9c05"} Jul 26 17:54:47.257469 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.257431 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-svrqf"] Jul 26 17:54:47.261368 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.261212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.263976 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.263947 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zvhsc\"" Jul 26 17:54:47.268803 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.268762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-svrqf"] Jul 26 17:54:47.415215 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.415176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.415413 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.415270 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwpm\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-kube-api-access-cmwpm\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.516230 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.516146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwpm\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-kube-api-access-cmwpm\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.516230 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.516226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.525933 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.525902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwpm\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-kube-api-access-cmwpm\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.526146 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.526119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9eb118df-8af0-4d14-aaf1-f0bfa40cc448-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-svrqf\" (UID: \"9eb118df-8af0-4d14-aaf1-f0bfa40cc448\") " pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:47.574766 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:47.574723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:49.181977 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.181954 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-svrqf"] Jul 26 17:54:49.183644 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:54:49.183614 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb118df_8af0_4d14_aaf1_f0bfa40cc448.slice/crio-31285841008adc2a1e47cc4899a574b4a5bc24b489aeb4794c38e721c3965b55 WatchSource:0}: Error finding container 31285841008adc2a1e47cc4899a574b4a5bc24b489aeb4794c38e721c3965b55: Status 404 returned error can't find the container with id 31285841008adc2a1e47cc4899a574b4a5bc24b489aeb4794c38e721c3965b55 Jul 26 17:54:49.803613 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.803571 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" event={"ID":"9eb118df-8af0-4d14-aaf1-f0bfa40cc448","Type":"ContainerStarted","Data":"77ccabc5384fb531b664b3fa6900144e83bf090a2f3778a482a0cef4279839c2"} Jul 26 17:54:49.803613 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.803625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" event={"ID":"9eb118df-8af0-4d14-aaf1-f0bfa40cc448","Type":"ContainerStarted","Data":"31285841008adc2a1e47cc4899a574b4a5bc24b489aeb4794c38e721c3965b55"} Jul 26 17:54:49.803912 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.803712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:54:49.805022 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.805000 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-npx6t" event={"ID":"d5a7ca1f-e921-4988-a2e9-b64635b29cd0","Type":"ContainerStarted","Data":"7239c5a28e4995261f4248ffae1159121f5798e5d7becbd2441f70059b74b902"} Jul 26 17:54:49.820066 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.820021 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" podStartSLOduration=2.820005751 podStartE2EDuration="2.820005751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:54:49.818671613 +0000 UTC m=+320.217700669" watchObservedRunningTime="2026-07-26 17:54:49.820005751 +0000 UTC m=+320.219034796" Jul 26 17:54:49.834450 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:49.834392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-npx6t" podStartSLOduration=1.708019629 podStartE2EDuration="4.834377838s" totalImagesPullingTime="3.126358209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:54:49.833498706 +0000 UTC m=+320.232527752" watchObservedRunningTime="2026-07-26 17:54:49.834377838 +0000 UTC m=+320.233406882" Jul 26 17:54:55.811539 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:54:55.811508 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-svrqf" Jul 26 17:55:08.137298 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.137261 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v"] Jul 26 17:55:08.140740 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.140719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.145331 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.145303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 17:55:08.145462 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.145346 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 17:55:08.145538 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.145486 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 17:55:08.145840 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.145823 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ml4x7\"" Jul 26 17:55:08.147312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.147292 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 17:55:08.177983 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.177950 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v"] Jul 26 17:55:08.181258 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.181231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlg2q\" (UniqueName: \"kubernetes.io/projected/476a8e69-43a2-4297-b3c5-61bc079361f4-kube-api-access-mlg2q\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.181389 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.181266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.181389 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.181306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-tmp\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.181389 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.181329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.181389 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.181357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-manifests\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282279 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-manifests\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg2q\" (UniqueName: \"kubernetes.io/projected/476a8e69-43a2-4297-b3c5-61bc079361f4-kube-api-access-mlg2q\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-tmp\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.282734 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.282692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-manifests\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.284676 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.284655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476a8e69-43a2-4297-b3c5-61bc079361f4-tmp\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.284932 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.284914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.285029 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.285010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/476a8e69-43a2-4297-b3c5-61bc079361f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.303302 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.303266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg2q\" (UniqueName: \"kubernetes.io/projected/476a8e69-43a2-4297-b3c5-61bc079361f4-kube-api-access-mlg2q\") pod \"opendatahub-operator-controller-manager-7b79b7df6c-v7h2v\" (UID: \"476a8e69-43a2-4297-b3c5-61bc079361f4\") " pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.451503 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.451411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:08.596528 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.596473 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v"] Jul 26 17:55:08.599031 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:55:08.599000 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476a8e69_43a2_4297_b3c5_61bc079361f4.slice/crio-85983791ab4383ae6c023134f2588c497cba4330b945616e86e2eb51d3efdd4f WatchSource:0}: Error finding container 85983791ab4383ae6c023134f2588c497cba4330b945616e86e2eb51d3efdd4f: Status 404 returned error can't find the container with id 85983791ab4383ae6c023134f2588c497cba4330b945616e86e2eb51d3efdd4f Jul 26 17:55:08.881738 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:08.881701 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" event={"ID":"476a8e69-43a2-4297-b3c5-61bc079361f4","Type":"ContainerStarted","Data":"85983791ab4383ae6c023134f2588c497cba4330b945616e86e2eb51d3efdd4f"} Jul 26 17:55:11.901493 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:11.901452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" event={"ID":"476a8e69-43a2-4297-b3c5-61bc079361f4","Type":"ContainerStarted","Data":"ab4eec40f106d6b21f95a2ad257607eed9c03b5a367505dccf862ae32d63c185"} Jul 26 17:55:12.906661 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:12.906625 2575 generic.go:356] "Generic (PLEG): container finished" podID="476a8e69-43a2-4297-b3c5-61bc079361f4" containerID="ab4eec40f106d6b21f95a2ad257607eed9c03b5a367505dccf862ae32d63c185" exitCode=0 Jul 26 17:55:12.907111 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:12.906703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" event={"ID":"476a8e69-43a2-4297-b3c5-61bc079361f4","Type":"ContainerDied","Data":"ab4eec40f106d6b21f95a2ad257607eed9c03b5a367505dccf862ae32d63c185"} Jul 26 17:55:13.912610 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:13.912575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" event={"ID":"476a8e69-43a2-4297-b3c5-61bc079361f4","Type":"ContainerStarted","Data":"61701ee8b46444ad6a84e65406116e3061296b48106195352c020ce662541f9c"} Jul 26 17:55:13.913022 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:13.912689 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:13.934921 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:13.934859 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" podStartSLOduration=1.550106525 podStartE2EDuration="5.934836364s" totalImagesPullingTime="3.309963143s" totalInitContainerRuntime="1.074766696s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:55:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:55:13.933470391 +0000 UTC m=+344.332499436" watchObservedRunningTime="2026-07-26 17:55:13.934836364 +0000 UTC m=+344.333865427" Jul 26 17:55:24.920863 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:24.920764 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b79b7df6c-v7h2v" Jul 26 17:55:36.717549 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.717515 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb"] Jul 26 17:55:36.722012 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.721991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.725264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.725213 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 17:55:36.725555 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.725537 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 17:55:36.725854 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.725836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dp88m\"" Jul 26 17:55:36.726410 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.726392 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 17:55:36.743408 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.743372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb"] Jul 26 17:55:36.831521 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831702 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831702 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831702 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831702 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831953 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831953 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8299eeda-2431-4c63-a25a-b0ef54e83123-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831953 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.831953 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.831850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2ll\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-kube-api-access-nw2ll\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933075 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933264 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8299eeda-2431-4c63-a25a-b0ef54e83123-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933566 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933566 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2ll\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-kube-api-access-nw2ll\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933566 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933566 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933779 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.933888 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.933841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.934097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.934078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8299eeda-2431-4c63-a25a-b0ef54e83123-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.935500 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.935480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.935756 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.935739 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.941382 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.941358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:36.941646 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:36.941627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2ll\" (UniqueName: \"kubernetes.io/projected/8299eeda-2431-4c63-a25a-b0ef54e83123-kube-api-access-nw2ll\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb\" (UID: \"8299eeda-2431-4c63-a25a-b0ef54e83123\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:37.033886 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:37.033773 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:37.208830 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:37.208780 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb"] Jul 26 17:55:37.214257 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:55:37.214234 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8299eeda_2431_4c63_a25a_b0ef54e83123.slice/crio-5fc89fe73c27c2d66186888d26c3735ec2915ddc0df11f4cf74b0003eff14460 WatchSource:0}: Error finding container 5fc89fe73c27c2d66186888d26c3735ec2915ddc0df11f4cf74b0003eff14460: Status 404 returned error can't find the container with id 5fc89fe73c27c2d66186888d26c3735ec2915ddc0df11f4cf74b0003eff14460 Jul 26 17:55:38.021959 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:38.021912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" event={"ID":"8299eeda-2431-4c63-a25a-b0ef54e83123","Type":"ContainerStarted","Data":"5fc89fe73c27c2d66186888d26c3735ec2915ddc0df11f4cf74b0003eff14460"} Jul 26 17:55:39.161024 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.160988 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x"] Jul 26 17:55:39.164388 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.163813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.169353 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169286 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 17:55:39.169495 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169375 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 26 17:55:39.169576 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169546 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 26 17:55:39.169710 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169663 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6tdb6\"" Jul 26 17:55:39.169781 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169772 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 17:55:39.169885 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.169816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 26 17:55:39.179657 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.179559 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x"] Jul 26 17:55:39.254856 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.254815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de976e4e-544b-45ac-824d-7378b74741ea-manager-config\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.255035 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.254884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.255035 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.254919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-metrics-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.255035 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.254969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659lk\" (UniqueName: \"kubernetes.io/projected/de976e4e-544b-45ac-824d-7378b74741ea-kube-api-access-659lk\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.274761 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.274729 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz"] Jul 26 17:55:39.277051 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.277035 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.289693 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.289662 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 26 17:55:39.289932 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.289919 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fq6ws\"" Jul 26 17:55:39.291561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.291540 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v"] Jul 26 17:55:39.293671 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.293655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.297032 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.296972 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-j9v6q\"" Jul 26 17:55:39.297144 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.297105 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 26 17:55:39.306625 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.306595 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz"] Jul 26 17:55:39.322641 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.322609 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v"] Jul 26 17:55:39.356261 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.356220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659lk\" (UniqueName: \"kubernetes.io/projected/de976e4e-544b-45ac-824d-7378b74741ea-kube-api-access-659lk\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.356441 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.356356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de976e4e-544b-45ac-824d-7378b74741ea-manager-config\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.356441 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.356405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.356572 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.356442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-metrics-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.357267 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.357241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/de976e4e-544b-45ac-824d-7378b74741ea-manager-config\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.359267 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.359244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.359375 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.359297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/de976e4e-544b-45ac-824d-7378b74741ea-metrics-cert\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.371304 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.371276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-659lk\" (UniqueName: \"kubernetes.io/projected/de976e4e-544b-45ac-824d-7378b74741ea-kube-api-access-659lk\") pod \"lws-controller-manager-fdddb58cf-t946x\" (UID: \"de976e4e-544b-45ac-824d-7378b74741ea\") " pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.457247 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.457161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/158ef9f0-f1f5-4345-b527-5790520d5b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.457430 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.457337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2xz\" (UniqueName: \"kubernetes.io/projected/158ef9f0-f1f5-4345-b527-5790520d5b04-kube-api-access-tt2xz\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.457430 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.457387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jc6p\" (UniqueName: \"kubernetes.io/projected/94e3d136-7f69-457b-82ea-1264ec5c2d1f-kube-api-access-8jc6p\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.457550 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.457444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.477470 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.477418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.558222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/158ef9f0-f1f5-4345-b527-5790520d5b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.558324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2xz\" (UniqueName: \"kubernetes.io/projected/158ef9f0-f1f5-4345-b527-5790520d5b04-kube-api-access-tt2xz\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.558355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jc6p\" (UniqueName: \"kubernetes.io/projected/94e3d136-7f69-457b-82ea-1264ec5c2d1f-kube-api-access-8jc6p\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.558378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:55:39.558504 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:55:39.558565 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert podName:94e3d136-7f69-457b-82ea-1264ec5c2d1f nodeName:}" failed. No retries permitted until 2026-07-26 17:55:40.058544014 +0000 UTC m=+370.457573037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert") pod "kserve-controller-manager-d6f4b65b8-ppz9v" (UID: "94e3d136-7f69-457b-82ea-1264ec5c2d1f") : secret "kserve-webhook-server-cert" not found Jul 26 17:55:39.567228 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.565592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/158ef9f0-f1f5-4345-b527-5790520d5b04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.579412 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.579349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jc6p\" (UniqueName: \"kubernetes.io/projected/94e3d136-7f69-457b-82ea-1264ec5c2d1f-kube-api-access-8jc6p\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:39.586862 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.583608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2xz\" (UniqueName: \"kubernetes.io/projected/158ef9f0-f1f5-4345-b527-5790520d5b04-kube-api-access-tt2xz\") pod \"llmisvc-controller-manager-777c6bd7d5-8r2wz\" (UID: \"158ef9f0-f1f5-4345-b527-5790520d5b04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.586862 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.586637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:39.729498 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:55:39.729374 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde976e4e_544b_45ac_824d_7378b74741ea.slice/crio-33c45b75341d9e587acd5a3b095730a8c86685a629496c71fb9be99e08312f6d WatchSource:0}: Error finding container 33c45b75341d9e587acd5a3b095730a8c86685a629496c71fb9be99e08312f6d: Status 404 returned error can't find the container with id 33c45b75341d9e587acd5a3b095730a8c86685a629496c71fb9be99e08312f6d Jul 26 17:55:39.729498 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.729391 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x"] Jul 26 17:55:39.798157 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:39.798016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz"] Jul 26 17:55:39.800842 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:55:39.800808 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod158ef9f0_f1f5_4345_b527_5790520d5b04.slice/crio-fc6a9d4303b3086701b97310d159d2c847157f5fc243db5399d25b7266badbb8 WatchSource:0}: Error finding container fc6a9d4303b3086701b97310d159d2c847157f5fc243db5399d25b7266badbb8: Status 404 returned error can't find the container with id fc6a9d4303b3086701b97310d159d2c847157f5fc243db5399d25b7266badbb8 Jul 26 17:55:40.035108 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.035010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" event={"ID":"158ef9f0-f1f5-4345-b527-5790520d5b04","Type":"ContainerStarted","Data":"fc6a9d4303b3086701b97310d159d2c847157f5fc243db5399d25b7266badbb8"} Jul 26 17:55:40.036455 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.036421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" event={"ID":"de976e4e-544b-45ac-824d-7378b74741ea","Type":"ContainerStarted","Data":"33c45b75341d9e587acd5a3b095730a8c86685a629496c71fb9be99e08312f6d"} Jul 26 17:55:40.064648 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.064607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:40.067462 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.067435 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94e3d136-7f69-457b-82ea-1264ec5c2d1f-cert\") pod \"kserve-controller-manager-d6f4b65b8-ppz9v\" (UID: \"94e3d136-7f69-457b-82ea-1264ec5c2d1f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:40.209565 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.209535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:40.927123 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.927071 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:55:40.927242 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.927169 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:55:40.927242 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:40.927207 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:55:41.018878 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:41.018846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v"] Jul 26 17:55:41.025634 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:55:41.025598 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e3d136_7f69_457b_82ea_1264ec5c2d1f.slice/crio-b0deca9be30463cd9df691a66fcbd052163a870032ee9dbca14624d9c3556dd9 WatchSource:0}: Error finding container b0deca9be30463cd9df691a66fcbd052163a870032ee9dbca14624d9c3556dd9: Status 404 returned error can't find the container with id b0deca9be30463cd9df691a66fcbd052163a870032ee9dbca14624d9c3556dd9 Jul 26 17:55:41.047718 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:41.047398 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" event={"ID":"94e3d136-7f69-457b-82ea-1264ec5c2d1f","Type":"ContainerStarted","Data":"b0deca9be30463cd9df691a66fcbd052163a870032ee9dbca14624d9c3556dd9"} Jul 26 17:55:41.051744 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:41.051706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" event={"ID":"8299eeda-2431-4c63-a25a-b0ef54e83123","Type":"ContainerStarted","Data":"45146c08dceaeb3c8810dfaf83a54859b11591d769a21c2477bed6c32d162691"} Jul 26 17:55:41.081224 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:41.080397 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" podStartSLOduration=1.369840218 podStartE2EDuration="5.080371234s" totalImagesPullingTime="3.710531016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:55:41.079017746 +0000 UTC m=+371.478046792" watchObservedRunningTime="2026-07-26 17:55:41.080371234 +0000 UTC m=+371.479400280" Jul 26 17:55:42.034876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:42.034840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:42.040831 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:42.040585 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:42.057859 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:42.057774 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:42.059039 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:42.059006 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb" Jul 26 17:55:45.076773 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.076731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" event={"ID":"94e3d136-7f69-457b-82ea-1264ec5c2d1f","Type":"ContainerStarted","Data":"350875572d480bb40e956672796640bf9ca7a4d8ab1e7ef42606cb641afbf07d"} Jul 26 17:55:45.077255 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.076816 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:55:45.078653 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.078621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" event={"ID":"de976e4e-544b-45ac-824d-7378b74741ea","Type":"ContainerStarted","Data":"1db89d64f074f9e5746f8957f4ad9e723a3fab5d0204984ee99d30edd0eb0ffa"} Jul 26 17:55:45.078818 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.078731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:55:45.080155 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.080136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" event={"ID":"158ef9f0-f1f5-4345-b527-5790520d5b04","Type":"ContainerStarted","Data":"ac3b9807c1113f8cabd1f25eb35195364d1b3b5649828d78dabbe10093f7a223"} Jul 26 17:55:45.080279 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.080262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:55:45.094828 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.094760 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" podStartSLOduration=2.629770386 podStartE2EDuration="6.094744593s" totalImagesPullingTime="3.464974207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:55:45.093745208 +0000 UTC m=+375.492774255" watchObservedRunningTime="2026-07-26 17:55:45.094744593 +0000 UTC m=+375.493773637" Jul 26 17:55:45.110823 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.110746 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" podStartSLOduration=1.350655157 podStartE2EDuration="6.110730758s" totalImagesPullingTime="4.760075601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:55:45.109988613 +0000 UTC m=+375.509017659" watchObservedRunningTime="2026-07-26 17:55:45.110730758 +0000 UTC m=+375.509759804" Jul 26 17:55:45.127625 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:45.127575 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" podStartSLOduration=1.432238156 podStartE2EDuration="6.127558078s" totalImagesPullingTime="4.695319922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:55:45.126305689 +0000 UTC m=+375.525334733" watchObservedRunningTime="2026-07-26 17:55:45.127558078 +0000 UTC m=+375.526587122" Jul 26 17:55:56.087978 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:55:56.087945 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fdddb58cf-t946x" Jul 26 17:56:16.087934 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:16.087904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8r2wz" Jul 26 17:56:16.090722 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:16.090696 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ppz9v" Jul 26 17:56:30.427942 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.427902 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6w7ls"] Jul 26 17:56:30.430157 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.430141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:30.432722 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.432702 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 17:56:30.433711 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.433696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-bvchv\"" Jul 26 17:56:30.433781 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.433741 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 17:56:30.438689 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.438664 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6w7ls"] Jul 26 17:56:30.610948 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.610908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqg7\" (UniqueName: \"kubernetes.io/projected/9bd37f86-de00-4c07-b85d-3dec6de95d5c-kube-api-access-gtqg7\") pod \"kuadrant-operator-catalog-6w7ls\" (UID: \"9bd37f86-de00-4c07-b85d-3dec6de95d5c\") " pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:30.712364 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.712265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqg7\" (UniqueName: \"kubernetes.io/projected/9bd37f86-de00-4c07-b85d-3dec6de95d5c-kube-api-access-gtqg7\") pod \"kuadrant-operator-catalog-6w7ls\" (UID: \"9bd37f86-de00-4c07-b85d-3dec6de95d5c\") " pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:30.720886 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.720855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqg7\" (UniqueName: \"kubernetes.io/projected/9bd37f86-de00-4c07-b85d-3dec6de95d5c-kube-api-access-gtqg7\") pod \"kuadrant-operator-catalog-6w7ls\" (UID: \"9bd37f86-de00-4c07-b85d-3dec6de95d5c\") " pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:30.740885 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.740845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:30.878935 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:30.878909 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6w7ls"] Jul 26 17:56:30.881438 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:56:30.881405 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd37f86_de00_4c07_b85d_3dec6de95d5c.slice/crio-d2e6f0098bb006c5d2b94a8baeb8c9815d2a834a17e8abb67321187f367457de WatchSource:0}: Error finding container d2e6f0098bb006c5d2b94a8baeb8c9815d2a834a17e8abb67321187f367457de: Status 404 returned error can't find the container with id d2e6f0098bb006c5d2b94a8baeb8c9815d2a834a17e8abb67321187f367457de Jul 26 17:56:31.292891 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:31.292855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" event={"ID":"9bd37f86-de00-4c07-b85d-3dec6de95d5c","Type":"ContainerStarted","Data":"d2e6f0098bb006c5d2b94a8baeb8c9815d2a834a17e8abb67321187f367457de"} Jul 26 17:56:33.302838 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:33.302727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" event={"ID":"9bd37f86-de00-4c07-b85d-3dec6de95d5c","Type":"ContainerStarted","Data":"0e149a9c66d356bb3b84a2dae9d1d8cd0f045008a744c87c4525b13c75d3c849"} Jul 26 17:56:33.318988 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:33.318927 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" podStartSLOduration=1.2521005170000001 podStartE2EDuration="3.318910784s" totalImagesPullingTime="2.066810267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:56:33.31696214 +0000 UTC m=+423.715991185" watchObservedRunningTime="2026-07-26 17:56:33.318910784 +0000 UTC m=+423.717939832" Jul 26 17:56:40.741671 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:40.741631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:40.742128 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:40.741705 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:40.763557 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:40.763525 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:56:41.360242 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:56:41.360214 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6w7ls" Jul 26 17:57:05.098949 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.098867 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt"] Jul 26 17:57:05.102436 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.102416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:05.106543 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.106508 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 26 17:57:05.106822 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.106805 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7kc72\"" Jul 26 17:57:05.125455 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.125421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt"] Jul 26 17:57:05.226932 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.226894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm8x\" (UniqueName: \"kubernetes.io/projected/00f88d38-30ba-4904-a23e-a47e6c85822e-kube-api-access-wkm8x\") pod \"dns-operator-controller-manager-864dd78c4c-hbblt\" (UID: \"00f88d38-30ba-4904-a23e-a47e6c85822e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:05.327534 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.327487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm8x\" (UniqueName: \"kubernetes.io/projected/00f88d38-30ba-4904-a23e-a47e6c85822e-kube-api-access-wkm8x\") pod \"dns-operator-controller-manager-864dd78c4c-hbblt\" (UID: \"00f88d38-30ba-4904-a23e-a47e6c85822e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:05.337690 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.337661 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm8x\" (UniqueName: \"kubernetes.io/projected/00f88d38-30ba-4904-a23e-a47e6c85822e-kube-api-access-wkm8x\") pod \"dns-operator-controller-manager-864dd78c4c-hbblt\" (UID: \"00f88d38-30ba-4904-a23e-a47e6c85822e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:05.412854 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.412825 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:05.560146 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:05.560119 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt"] Jul 26 17:57:05.562341 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:57:05.562313 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f88d38_30ba_4904_a23e_a47e6c85822e.slice/crio-f736e81505ea0c2f9ac3a3b0189b630a2c5641c01f5b6ea1233033ab773e2786 WatchSource:0}: Error finding container f736e81505ea0c2f9ac3a3b0189b630a2c5641c01f5b6ea1233033ab773e2786: Status 404 returned error can't find the container with id f736e81505ea0c2f9ac3a3b0189b630a2c5641c01f5b6ea1233033ab773e2786 Jul 26 17:57:06.456876 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:06.456837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" event={"ID":"00f88d38-30ba-4904-a23e-a47e6c85822e","Type":"ContainerStarted","Data":"f736e81505ea0c2f9ac3a3b0189b630a2c5641c01f5b6ea1233033ab773e2786"} Jul 26 17:57:08.474089 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.474054 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" event={"ID":"00f88d38-30ba-4904-a23e-a47e6c85822e","Type":"ContainerStarted","Data":"6b95bad200a3613a452169df81d52c3e2a446177effd4ab12ca8c6e530b9a7e3"} Jul 26 17:57:08.474477 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.474146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:08.494615 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.494562 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" podStartSLOduration=1.513431264 podStartE2EDuration="3.494547709s" totalImagesPullingTime="1.981116445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:57:08.49272306 +0000 UTC m=+458.891752111" watchObservedRunningTime="2026-07-26 17:57:08.494547709 +0000 UTC m=+458.893576755" Jul 26 17:57:08.646507 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.646469 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5"] Jul 26 17:57:08.649923 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.649907 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:08.652798 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.652767 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zthbc\"" Jul 26 17:57:08.673074 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.673039 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5"] Jul 26 17:57:08.759688 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.759605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkdd\" (UniqueName: \"kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd\") pod \"limitador-operator-controller-manager-859d7b78bd-5thg5\" (UID: \"73cf374f-0ea7-4722-8af2-3b88e90f41db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:08.861032 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.860994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkdd\" (UniqueName: \"kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd\") pod \"limitador-operator-controller-manager-859d7b78bd-5thg5\" (UID: \"73cf374f-0ea7-4722-8af2-3b88e90f41db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:08.871096 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.871068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkdd\" (UniqueName: \"kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd\") pod \"limitador-operator-controller-manager-859d7b78bd-5thg5\" (UID: \"73cf374f-0ea7-4722-8af2-3b88e90f41db\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:08.959725 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:08.959684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:09.097920 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:09.097895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5"] Jul 26 17:57:09.100757 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:57:09.100732 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cf374f_0ea7_4722_8af2_3b88e90f41db.slice/crio-0e7340a30e45083c91fd544f274106ec54797dd56d1e5f2a3da60d40ea5876df WatchSource:0}: Error finding container 0e7340a30e45083c91fd544f274106ec54797dd56d1e5f2a3da60d40ea5876df: Status 404 returned error can't find the container with id 0e7340a30e45083c91fd544f274106ec54797dd56d1e5f2a3da60d40ea5876df Jul 26 17:57:09.479912 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:09.479864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" event={"ID":"73cf374f-0ea7-4722-8af2-3b88e90f41db","Type":"ContainerStarted","Data":"0e7340a30e45083c91fd544f274106ec54797dd56d1e5f2a3da60d40ea5876df"} Jul 26 17:57:11.491994 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:11.491960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" event={"ID":"73cf374f-0ea7-4722-8af2-3b88e90f41db","Type":"ContainerStarted","Data":"b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7"} Jul 26 17:57:11.492383 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:11.492095 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:11.510963 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:11.510912 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" podStartSLOduration=2.061513449 podStartE2EDuration="3.510893801s" totalImagesPullingTime="1.449380352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:57:11.509585935 +0000 UTC m=+461.908614981" watchObservedRunningTime="2026-07-26 17:57:11.510893801 +0000 UTC m=+461.909922845" Jul 26 17:57:19.482944 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:19.482908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hbblt" Jul 26 17:57:22.500669 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:22.500636 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:31.311250 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.311216 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr"] Jul 26 17:57:31.315097 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.315072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.318484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.318463 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rqdcd\"" Jul 26 17:57:31.336437 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.336402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr"] Jul 26 17:57:31.357969 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.357934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxw5\" (UniqueName: \"kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.358150 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.357993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.459260 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.459216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.459437 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.459329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxw5\" (UniqueName: \"kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.459619 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.459596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.468142 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.468114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxw5\" (UniqueName: \"kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.626080 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.626045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:31.786535 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:31.786497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr"] Jul 26 17:57:31.789246 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:57:31.789210 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ec3121_a1f7_4980_8227_46e5f3a19d56.slice/crio-15c859a9614339caae94810ff0732c00433a14e7b5d2a922d64c08a62e3b581e WatchSource:0}: Error finding container 15c859a9614339caae94810ff0732c00433a14e7b5d2a922d64c08a62e3b581e: Status 404 returned error can't find the container with id 15c859a9614339caae94810ff0732c00433a14e7b5d2a922d64c08a62e3b581e Jul 26 17:57:32.017527 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.017437 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr"] Jul 26 17:57:32.027412 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.027383 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr"] Jul 26 17:57:32.059331 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.059296 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5"] Jul 26 17:57:32.059530 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.059504 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" containerName="manager" containerID="cri-o://b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7" gracePeriod=2 Jul 26 17:57:32.066430 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.066403 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5"] Jul 26 17:57:32.073775 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.073745 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:32.074156 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.074141 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" containerName="manager" Jul 26 17:57:32.074199 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.074157 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" containerName="manager" Jul 26 17:57:32.074233 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.074224 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" containerName="manager" Jul 26 17:57:32.077266 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.077248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.079722 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:32.079690 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5thg5\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:32.098728 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.098682 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn"] Jul 26 17:57:32.102279 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.102253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:32.123452 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.123421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn"] Jul 26 17:57:32.142953 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:32.142919 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5thg5\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:32.153937 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.153907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:32.166272 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.166244 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.166272 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.166277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4mm\" (UniqueName: \"kubernetes.io/projected/fff2f629-14af-4d71-bc14-81a03f68682d-kube-api-access-qr4mm\") pod \"limitador-operator-controller-manager-859d7b78bd-r5gfn\" (UID: \"fff2f629-14af-4d71-bc14-81a03f68682d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:32.166469 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.166326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb5g\" (UniqueName: \"kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.267521 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.267487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.267732 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.267527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4mm\" (UniqueName: \"kubernetes.io/projected/fff2f629-14af-4d71-bc14-81a03f68682d-kube-api-access-qr4mm\") pod \"limitador-operator-controller-manager-859d7b78bd-r5gfn\" (UID: \"fff2f629-14af-4d71-bc14-81a03f68682d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:32.267732 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.267566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb5g\" (UniqueName: \"kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.267974 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.267951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.279615 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.279584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb5g\" (UniqueName: \"kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v9ffd\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.282669 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.282641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4mm\" (UniqueName: \"kubernetes.io/projected/fff2f629-14af-4d71-bc14-81a03f68682d-kube-api-access-qr4mm\") pod \"limitador-operator-controller-manager-859d7b78bd-r5gfn\" (UID: \"fff2f629-14af-4d71-bc14-81a03f68682d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:32.299710 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.299683 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:32.368813 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.368756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zkdd\" (UniqueName: \"kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd\") pod \"73cf374f-0ea7-4722-8af2-3b88e90f41db\" (UID: \"73cf374f-0ea7-4722-8af2-3b88e90f41db\") " Jul 26 17:57:32.371114 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.371086 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd" (OuterVolumeSpecName: "kube-api-access-9zkdd") pod "73cf374f-0ea7-4722-8af2-3b88e90f41db" (UID: "73cf374f-0ea7-4722-8af2-3b88e90f41db"). InnerVolumeSpecName "kube-api-access-9zkdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:57:32.441140 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.441099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:32.447973 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.447943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:32.470178 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.470144 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zkdd\" (UniqueName: \"kubernetes.io/projected/73cf374f-0ea7-4722-8af2-3b88e90f41db-kube-api-access-9zkdd\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:57:32.601083 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.600955 2575 generic.go:356] "Generic (PLEG): container finished" podID="73cf374f-0ea7-4722-8af2-3b88e90f41db" containerID="b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7" exitCode=0 Jul 26 17:57:32.601083 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.601076 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5thg5" Jul 26 17:57:32.605074 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.604874 2575 scope.go:117] "RemoveContainer" containerID="b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7" Jul 26 17:57:32.630476 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.630326 2575 scope.go:117] "RemoveContainer" containerID="b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7" Jul 26 17:57:32.630761 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:32.630730 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7\": container with ID starting with b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7 not found: ID does not exist" containerID="b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7" Jul 26 17:57:32.630907 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.630774 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7"} err="failed to get container status \"b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7\": rpc error: code = NotFound desc = could not find container \"b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7\": container with ID starting with b0af5aaa9c5c6b30ecde117aabfe7618626c590f750122705cd8ac40c88559a7 not found: ID does not exist" Jul 26 17:57:32.653312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.653279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:32.655519 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:57:32.655488 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ee4247_158c_4a44_94c0_ea9f0ab39e6c.slice/crio-032a33799c20d6849a57331feac5d21589a42f10f13607590d0e80b0312d7cf7 WatchSource:0}: Error finding container 032a33799c20d6849a57331feac5d21589a42f10f13607590d0e80b0312d7cf7: Status 404 returned error can't find the container with id 032a33799c20d6849a57331feac5d21589a42f10f13607590d0e80b0312d7cf7 Jul 26 17:57:32.677568 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:32.677542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn"] Jul 26 17:57:32.680263 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:57:32.680231 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff2f629_14af_4d71_bc14_81a03f68682d.slice/crio-e1e85961f3de6814e41d73b7ef41a895db48f4e5a059dea49a845e1313a55b7f WatchSource:0}: Error finding container e1e85961f3de6814e41d73b7ef41a895db48f4e5a059dea49a845e1313a55b7f: Status 404 returned error can't find the container with id e1e85961f3de6814e41d73b7ef41a895db48f4e5a059dea49a845e1313a55b7f Jul 26 17:57:33.627230 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:33.627190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" event={"ID":"97ee4247-158c-4a44-94c0-ea9f0ab39e6c","Type":"ContainerStarted","Data":"032a33799c20d6849a57331feac5d21589a42f10f13607590d0e80b0312d7cf7"} Jul 26 17:57:33.629340 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:33.629305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" event={"ID":"fff2f629-14af-4d71-bc14-81a03f68682d","Type":"ContainerStarted","Data":"02f5ecaca388497d66421d4677d7a0082f87bc03eec4fd1600ebb2a1c16fd211"} Jul 26 17:57:33.629482 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:33.629428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" event={"ID":"fff2f629-14af-4d71-bc14-81a03f68682d","Type":"ContainerStarted","Data":"e1e85961f3de6814e41d73b7ef41a895db48f4e5a059dea49a845e1313a55b7f"} Jul 26 17:57:33.629580 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:33.629557 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:33.667271 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:33.667219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" podStartSLOduration=1.6671982650000001 podStartE2EDuration="1.667198265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:57:33.664993197 +0000 UTC m=+484.064022243" watchObservedRunningTime="2026-07-26 17:57:33.667198265 +0000 UTC m=+484.066227312" Jul 26 17:57:34.211492 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:34.211460 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73cf374f-0ea7-4722-8af2-3b88e90f41db" path="/var/lib/kubelet/pods/73cf374f-0ea7-4722-8af2-3b88e90f41db/volumes" Jul 26 17:57:36.649699 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.649642 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" containerName="manager" containerID="cri-o://e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a" gracePeriod=2 Jul 26 17:57:36.651737 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.651702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" event={"ID":"97ee4247-158c-4a44-94c0-ea9f0ab39e6c","Type":"ContainerStarted","Data":"7329f26506927ca7be91658495a45fcccb5386fd1d72c6c1205fc70338de7284"} Jul 26 17:57:36.651882 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.651855 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:36.678559 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.678508 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" podStartSLOduration=1.593358227 podStartE2EDuration="4.678492482s" totalImagesPullingTime="3.085134255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:57:36.677192398 +0000 UTC m=+487.076221467" watchObservedRunningTime="2026-07-26 17:57:36.678492482 +0000 UTC m=+487.077521527" Jul 26 17:57:36.889710 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.889685 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:36.892010 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:36.891982 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:36.914525 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.914442 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcxw5\" (UniqueName: \"kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5\") pod \"83ec3121-a1f7-4980-8227-46e5f3a19d56\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " Jul 26 17:57:36.914663 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.914554 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume\") pod \"83ec3121-a1f7-4980-8227-46e5f3a19d56\" (UID: \"83ec3121-a1f7-4980-8227-46e5f3a19d56\") " Jul 26 17:57:36.914859 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.914834 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "83ec3121-a1f7-4980-8227-46e5f3a19d56" (UID: "83ec3121-a1f7-4980-8227-46e5f3a19d56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:57:36.916470 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:36.916446 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5" (OuterVolumeSpecName: "kube-api-access-xcxw5") pod "83ec3121-a1f7-4980-8227-46e5f3a19d56" (UID: "83ec3121-a1f7-4980-8227-46e5f3a19d56"). InnerVolumeSpecName "kube-api-access-xcxw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:57:37.015724 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.015687 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxw5\" (UniqueName: \"kubernetes.io/projected/83ec3121-a1f7-4980-8227-46e5f3a19d56-kube-api-access-xcxw5\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:57:37.015724 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.015716 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/83ec3121-a1f7-4980-8227-46e5f3a19d56-extensions-socket-volume\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:57:37.658421 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.658385 2575 generic.go:356] "Generic (PLEG): container finished" podID="83ec3121-a1f7-4980-8227-46e5f3a19d56" containerID="e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a" exitCode=2 Jul 26 17:57:37.658852 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.658433 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:37.658852 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.658485 2575 scope.go:117] "RemoveContainer" containerID="e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a" Jul 26 17:57:37.660971 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:37.660943 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:37.667160 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.667144 2575 scope.go:117] "RemoveContainer" containerID="e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a" Jul 26 17:57:37.667398 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:37.667379 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a\": container with ID starting with e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a not found: ID does not exist" containerID="e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a" Jul 26 17:57:37.667458 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:37.667406 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a"} err="failed to get container status \"e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a\": rpc error: code = NotFound desc = could not find container \"e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a\": container with ID starting with e7f40a8038cc73e239518360e0cfb8053a8d5a212f14df67de8721385527290a not found: ID does not exist" Jul 26 17:57:37.668744 ip-10-0-138-219 kubenswrapper[2575]: E0726 17:57:37.668726 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-64rdr\" is forbidden: User \"system:node:ip-10-0-138-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-219.ec2.internal' and this object" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-64rdr" Jul 26 17:57:38.209306 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:38.209273 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" path="/var/lib/kubelet/pods/83ec3121-a1f7-4980-8227-46e5f3a19d56/volumes" Jul 26 17:57:44.451520 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.451480 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:44.451970 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.451726 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" podUID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" containerName="manager" containerID="cri-o://7329f26506927ca7be91658495a45fcccb5386fd1d72c6c1205fc70338de7284" gracePeriod=10 Jul 26 17:57:44.453620 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.453596 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:44.639886 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.639860 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r5gfn" Jul 26 17:57:44.695817 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.695773 2575 generic.go:356] "Generic (PLEG): container finished" podID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" containerID="7329f26506927ca7be91658495a45fcccb5386fd1d72c6c1205fc70338de7284" exitCode=0 Jul 26 17:57:44.695955 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.695853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" event={"ID":"97ee4247-158c-4a44-94c0-ea9f0ab39e6c","Type":"ContainerDied","Data":"7329f26506927ca7be91658495a45fcccb5386fd1d72c6c1205fc70338de7284"} Jul 26 17:57:44.715461 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.715385 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:44.780344 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.780309 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnb5g\" (UniqueName: \"kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g\") pod \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " Jul 26 17:57:44.780538 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.780406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume\") pod \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\" (UID: \"97ee4247-158c-4a44-94c0-ea9f0ab39e6c\") " Jul 26 17:57:44.780884 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.780854 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97ee4247-158c-4a44-94c0-ea9f0ab39e6c" (UID: "97ee4247-158c-4a44-94c0-ea9f0ab39e6c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 17:57:44.782376 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.782353 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g" (OuterVolumeSpecName: "kube-api-access-mnb5g") pod "97ee4247-158c-4a44-94c0-ea9f0ab39e6c" (UID: "97ee4247-158c-4a44-94c0-ea9f0ab39e6c"). InnerVolumeSpecName "kube-api-access-mnb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 17:57:44.882040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.881998 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb5g\" (UniqueName: \"kubernetes.io/projected/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-kube-api-access-mnb5g\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:57:44.882040 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:44.882034 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97ee4247-158c-4a44-94c0-ea9f0ab39e6c-extensions-socket-volume\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 17:57:45.702038 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:45.702003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" event={"ID":"97ee4247-158c-4a44-94c0-ea9f0ab39e6c","Type":"ContainerDied","Data":"032a33799c20d6849a57331feac5d21589a42f10f13607590d0e80b0312d7cf7"} Jul 26 17:57:45.702474 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:45.702066 2575 scope.go:117] "RemoveContainer" containerID="7329f26506927ca7be91658495a45fcccb5386fd1d72c6c1205fc70338de7284" Jul 26 17:57:45.702474 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:45.702023 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd" Jul 26 17:57:45.739112 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:45.739075 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:45.744421 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:45.744392 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v9ffd"] Jul 26 17:57:46.208985 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:57:46.208954 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" path="/var/lib/kubelet/pods/97ee4247-158c-4a44-94c0-ea9f0ab39e6c/volumes" Jul 26 17:58:00.793210 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.793175 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2"] Jul 26 17:58:00.793906 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.793867 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" containerName="manager" Jul 26 17:58:00.793906 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.793891 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" containerName="manager" Jul 26 17:58:00.794071 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.793925 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" containerName="manager" Jul 26 17:58:00.794071 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.793934 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" containerName="manager" Jul 26 17:58:00.794071 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.794035 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ec3121-a1f7-4980-8227-46e5f3a19d56" containerName="manager" Jul 26 17:58:00.794071 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.794049 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ee4247-158c-4a44-94c0-ea9f0ab39e6c" containerName="manager" Jul 26 17:58:00.797127 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.797107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.799951 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.799931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sxbw4\"" Jul 26 17:58:00.808556 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.808535 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2"] Jul 26 17:58:00.931844 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.931778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932014 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.931856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932014 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.931877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9r5\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-kube-api-access-fw9r5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932014 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.931897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932014 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.931986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.932028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.932048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.932077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:00.932148 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:00.932126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.032883 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.032849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.032897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.032926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.032965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033065 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.032993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033286 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033286 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033286 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9r5\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-kube-api-access-fw9r5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033286 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033480 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033480 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033388 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033657 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033699 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.033778 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.033740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.035536 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.035515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.035777 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.035759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.044470 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.044411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.044635 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.044615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9r5\" (UniqueName: \"kubernetes.io/projected/cec9cd99-7139-47b9-8aaf-f2ad972f6809-kube-api-access-fw9r5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-9nbc2\" (UID: \"cec9cd99-7139-47b9-8aaf-f2ad972f6809\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.107764 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.107731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:01.247055 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.247030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2"] Jul 26 17:58:01.248938 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:58:01.248910 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec9cd99_7139_47b9_8aaf_f2ad972f6809.slice/crio-241bbbe6a3aaf55751aa66fdf5c4c82f4020b324168d0eb3002b9d075cc42e0d WatchSource:0}: Error finding container 241bbbe6a3aaf55751aa66fdf5c4c82f4020b324168d0eb3002b9d075cc42e0d: Status 404 returned error can't find the container with id 241bbbe6a3aaf55751aa66fdf5c4c82f4020b324168d0eb3002b9d075cc42e0d Jul 26 17:58:01.250988 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.250956 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:58:01.251071 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.251019 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:58:01.251109 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.251069 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:58:01.782076 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.782037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" event={"ID":"cec9cd99-7139-47b9-8aaf-f2ad972f6809","Type":"ContainerStarted","Data":"eefcbb1a851389791f23d04e79c4b5a7e50078c0c010f1b4b81789c448cadf41"} Jul 26 17:58:01.782280 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.782090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" event={"ID":"cec9cd99-7139-47b9-8aaf-f2ad972f6809","Type":"ContainerStarted","Data":"241bbbe6a3aaf55751aa66fdf5c4c82f4020b324168d0eb3002b9d075cc42e0d"} Jul 26 17:58:01.803220 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:01.803165 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podStartSLOduration=1.803148415 podStartE2EDuration="1.803148415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:58:01.801186177 +0000 UTC m=+512.200215264" watchObservedRunningTime="2026-07-26 17:58:01.803148415 +0000 UTC m=+512.202177459" Jul 26 17:58:02.108132 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:02.108096 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:02.109443 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:02.109417 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9nbc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 26 17:58:02.109561 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:02.109475 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podUID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 26 17:58:03.112470 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:03.112437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:03.791766 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:03.791735 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:03.792816 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:03.792777 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:58:15.550138 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.550105 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:58:15.553572 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.553548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.556202 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.556179 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 26 17:58:15.556377 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.556359 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9sdhc\"" Jul 26 17:58:15.563312 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.563289 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:58:15.593777 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.593739 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:58:15.667978 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.667936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-config-file\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.668149 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.668040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m7r9\" (UniqueName: \"kubernetes.io/projected/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-kube-api-access-5m7r9\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.769310 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.769277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-config-file\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.769477 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.769344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7r9\" (UniqueName: \"kubernetes.io/projected/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-kube-api-access-5m7r9\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.769930 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.769911 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-config-file\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.777333 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.777302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7r9\" (UniqueName: \"kubernetes.io/projected/bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f-kube-api-access-5m7r9\") pod \"limitador-limitador-54d6d988d5-q7zs9\" (UID: \"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.865554 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.865519 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:15.999132 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:15.999105 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:58:16.001049 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:58:16.001018 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc0bd0a_2cf4_4d6f_80fb_534dfb20b55f.slice/crio-2e4c72a9af128f1187504760b87fceaa6821ce1886f4f00e2b3098f70ae65759 WatchSource:0}: Error finding container 2e4c72a9af128f1187504760b87fceaa6821ce1886f4f00e2b3098f70ae65759: Status 404 returned error can't find the container with id 2e4c72a9af128f1187504760b87fceaa6821ce1886f4f00e2b3098f70ae65759 Jul 26 17:58:16.861564 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:16.861520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" event={"ID":"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f","Type":"ContainerStarted","Data":"2e4c72a9af128f1187504760b87fceaa6821ce1886f4f00e2b3098f70ae65759"} Jul 26 17:58:18.873328 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:18.873290 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" event={"ID":"bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f","Type":"ContainerStarted","Data":"1ff61fa2444135520df7339fabb04fc8d69acd135ce974a3a55f33666c69b229"} Jul 26 17:58:18.873740 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:18.873399 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:58:18.890260 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:18.890207 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" podStartSLOduration=1.510143497 podStartE2EDuration="3.890190769s" totalImagesPullingTime="2.380047272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:58:18.888048534 +0000 UTC m=+529.287077580" watchObservedRunningTime="2026-07-26 17:58:18.890190769 +0000 UTC m=+529.289219814" Jul 26 17:58:29.879667 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:58:29.879572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7zs9" Jul 26 17:59:31.433175 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:31.433139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:59:39.265145 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.265053 2575 generic.go:356] "Generic (PLEG): container finished" podID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerID="eefcbb1a851389791f23d04e79c4b5a7e50078c0c010f1b4b81789c448cadf41" exitCode=0 Jul 26 17:59:39.265145 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.265127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" event={"ID":"cec9cd99-7139-47b9-8aaf-f2ad972f6809","Type":"ContainerDied","Data":"eefcbb1a851389791f23d04e79c4b5a7e50078c0c010f1b4b81789c448cadf41"} Jul 26 17:59:39.265553 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.265498 2575 scope.go:117] "RemoveContainer" containerID="eefcbb1a851389791f23d04e79c4b5a7e50078c0c010f1b4b81789c448cadf41" Jul 26 17:59:39.268940 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.268905 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:59:39.269089 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.268993 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:59:39.269089 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:39.269039 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 17:59:40.271604 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:40.271574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" event={"ID":"cec9cd99-7139-47b9-8aaf-f2ad972f6809","Type":"ContainerStarted","Data":"8e839a029570a26ddb1d2fac2f2badea5f8703c74a18795527c4ea9fe9b11f22"} Jul 26 17:59:41.108986 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:41.108946 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:59:41.109186 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:41.109162 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9nbc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 26 17:59:41.109236 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:41.109216 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podUID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 26 17:59:42.108358 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:42.108319 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9nbc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 26 17:59:42.108758 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:42.108376 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podUID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 26 17:59:43.108596 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:43.108556 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9nbc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 26 17:59:43.109102 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:43.108620 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podUID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 26 17:59:44.108666 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:44.108628 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-9nbc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 26 17:59:44.109082 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:44.108685 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" podUID="cec9cd99-7139-47b9-8aaf-f2ad972f6809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 26 17:59:45.113377 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:45.113344 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:59:45.113804 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:45.113777 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:59:45.115190 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:45.115167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-9nbc2" Jul 26 17:59:57.581518 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.581434 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg"] Jul 26 17:59:57.585624 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.585602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.589753 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.589723 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 26 17:59:57.590020 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.590000 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 17:59:57.590174 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.590150 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8klrd\"" Jul 26 17:59:57.590304 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.590188 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 17:59:57.592367 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.592338 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg"] Jul 26 17:59:57.731986 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.731949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.732172 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.732000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9335a-d820-4dfa-b2a2-821ae96daaed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.732172 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.732119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.732308 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.732178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.732308 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.732252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.732308 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.732293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hzr\" (UniqueName: \"kubernetes.io/projected/72b9335a-d820-4dfa-b2a2-821ae96daaed-kube-api-access-p7hzr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.832868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9335a-d820-4dfa-b2a2-821ae96daaed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.832868 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833078 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833078 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833078 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hzr\" (UniqueName: \"kubernetes.io/projected/72b9335a-d820-4dfa-b2a2-821ae96daaed-kube-api-access-p7hzr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833078 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.832983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833322 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.833299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833395 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.833344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.833432 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.833399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.835461 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.835429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b9335a-d820-4dfa-b2a2-821ae96daaed-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.835580 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.835522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b9335a-d820-4dfa-b2a2-821ae96daaed-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.841484 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.841462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hzr\" (UniqueName: \"kubernetes.io/projected/72b9335a-d820-4dfa-b2a2-821ae96daaed-kube-api-access-p7hzr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg\" (UID: \"72b9335a-d820-4dfa-b2a2-821ae96daaed\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:57.898586 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:57.898545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 17:59:58.035257 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:58.035227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg"] Jul 26 17:59:58.037147 ip-10-0-138-219 kubenswrapper[2575]: W0726 17:59:58.037102 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b9335a_d820_4dfa_b2a2_821ae96daaed.slice/crio-6a37f1b2d6c9369e3d994a56b75dd6e9443b2853875ee94b9a5e57b70560976a WatchSource:0}: Error finding container 6a37f1b2d6c9369e3d994a56b75dd6e9443b2853875ee94b9a5e57b70560976a: Status 404 returned error can't find the container with id 6a37f1b2d6c9369e3d994a56b75dd6e9443b2853875ee94b9a5e57b70560976a Jul 26 17:59:58.039010 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:58.038986 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 17:59:58.084600 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:58.084516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 17:59:58.361395 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:58.361354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" event={"ID":"72b9335a-d820-4dfa-b2a2-821ae96daaed","Type":"ContainerStarted","Data":"6a37f1b2d6c9369e3d994a56b75dd6e9443b2853875ee94b9a5e57b70560976a"} Jul 26 17:59:59.367548 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:59.367505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" event={"ID":"72b9335a-d820-4dfa-b2a2-821ae96daaed","Type":"ContainerStarted","Data":"2089f30f4864188ed192c794a46e6b5b13ae4a30ac55282e2dcd4a17446c3c42"} Jul 26 17:59:59.385057 ip-10-0-138-219 kubenswrapper[2575]: I0726 17:59:59.385004 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" podStartSLOduration=1.34680538 podStartE2EDuration="2.384986617s" totalImagesPullingTime="1.038181237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 17:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 17:59:59.38401568 +0000 UTC m=+629.783044726" watchObservedRunningTime="2026-07-26 17:59:59.384986617 +0000 UTC m=+629.784015661" Jul 26 18:00:07.899210 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:07.899170 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 18:00:07.899719 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:07.899251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 18:00:07.912051 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:07.912025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 18:00:08.431041 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:08.431010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg" Jul 26 18:00:10.170114 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:10.170077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:00:29.968256 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:29.968221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:00:36.161354 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:36.161312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:00:43.871837 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:43.871797 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:00:53.332365 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.332331 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462"] Jul 26 18:00:53.335860 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.335840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.338799 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.338762 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 26 18:00:53.345626 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.345603 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462"] Jul 26 18:00:53.448411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf0636d-9254-4997-a939-6ee6a74b27d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.448590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.448590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.448590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.448590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnr9v\" (UniqueName: \"kubernetes.io/projected/bdf0636d-9254-4997-a939-6ee6a74b27d0-kube-api-access-bnr9v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.448590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.448537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.549855 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.549778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnr9v\" (UniqueName: \"kubernetes.io/projected/bdf0636d-9254-4997-a939-6ee6a74b27d0-kube-api-access-bnr9v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.549871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.549923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf0636d-9254-4997-a939-6ee6a74b27d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.549980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550230 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.550024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550230 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.550049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.550331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550415 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.550350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.550415 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.550383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.552304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.552267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf0636d-9254-4997-a939-6ee6a74b27d0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.552426 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.552409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf0636d-9254-4997-a939-6ee6a74b27d0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.558878 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.558855 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnr9v\" (UniqueName: \"kubernetes.io/projected/bdf0636d-9254-4997-a939-6ee6a74b27d0-kube-api-access-bnr9v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5s462\" (UID: \"bdf0636d-9254-4997-a939-6ee6a74b27d0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.645502 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.645456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:00:53.785506 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:53.785353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462"] Jul 26 18:00:53.787881 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:00:53.787845 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdf0636d_9254_4997_a939_6ee6a74b27d0.slice/crio-193fcbdd66c19ee33071717160c7e149be2cf2e73dc02a23e076d1ed2154d940 WatchSource:0}: Error finding container 193fcbdd66c19ee33071717160c7e149be2cf2e73dc02a23e076d1ed2154d940: Status 404 returned error can't find the container with id 193fcbdd66c19ee33071717160c7e149be2cf2e73dc02a23e076d1ed2154d940 Jul 26 18:00:54.655028 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:54.654988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" event={"ID":"bdf0636d-9254-4997-a939-6ee6a74b27d0","Type":"ContainerStarted","Data":"e90b1be98b00574fc35cfbc6b4d29429a523d0e062089d07e5384b88900384b0"} Jul 26 18:00:54.655028 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:54.655030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" event={"ID":"bdf0636d-9254-4997-a939-6ee6a74b27d0","Type":"ContainerStarted","Data":"193fcbdd66c19ee33071717160c7e149be2cf2e73dc02a23e076d1ed2154d940"} Jul 26 18:00:54.675851 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:54.675781 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" podStartSLOduration=1.514127828 podStartE2EDuration="1.675764806s" totalImagesPullingTime="161.636978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:00:54.673990057 +0000 UTC m=+685.073019103" watchObservedRunningTime="2026-07-26 18:00:54.675764806 +0000 UTC m=+685.074793850" Jul 26 18:00:55.872761 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:00:55.872731 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:01:03.645856 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:01:03.645785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:01:03.646326 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:01:03.645879 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:01:03.658364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:01:03.658334 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:01:03.710394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:01:03.710360 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5s462" Jul 26 18:02:44.289722 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:02:44.289683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:02:53.668699 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:02:53.668616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:03:04.805771 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:03:04.805727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:04:48.776216 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:04:48.776179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:04:56.884305 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:04:56.884269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:05:28.771008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:05:28.770968 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:05:38.777733 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:05:38.777699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:05:47.773590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:05:47.773555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:05:58.772810 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:05:58.772721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:06:08.180398 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:06:08.180367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:06:23.872060 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:06:23.872021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:06:30.368001 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:06:30.367968 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:06:39.363370 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:06:39.363328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:07:01.092665 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:07:01.092630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:07:12.580497 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:07:12.580454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:07:37.786191 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:07:37.786101 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:07:48.513255 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:07:48.513218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:08:03.281292 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:08:03.281250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:08:13.668344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:08:13.668305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:08:23.562677 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:08:23.562639 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:08:32.263722 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:08:32.263680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:08:41.782581 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:08:41.782540 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:10:30.273712 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:10:30.273626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:10:39.167203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:10:39.167159 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:10:51.869534 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:10:51.869485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:03.570281 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:03.570247 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:13.170448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:13.170408 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:24.865190 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:24.865150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:34.562261 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:34.562221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:44.665927 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:44.665886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:11:55.276009 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:11:55.275917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:12:04.566869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:04.566835 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:12:14.973308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:14.973270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:12:36.473761 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:36.473718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:12:46.509953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:46.509918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:12:52.443072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.443038 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:12:52.446711 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.446693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.450099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.450077 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-09f1ce9d-kserve-self-signed-certs\"" Jul 26 18:12:52.477575 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.477545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:12:52.536662 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.536662 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.536943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.536943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.536943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.536943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.536849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq6w\" (UniqueName: \"kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.637610 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.637830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.637830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.637830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.637830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.638061 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.637832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq6w\" (UniqueName: \"kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.638119 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.638061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.638332 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.638295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.638460 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.638406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.640229 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.640206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.640568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.640228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.647129 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.647105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq6w\" (UniqueName: \"kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w\") pod \"e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.756488 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.756407 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:12:52.907544 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:52.907517 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:12:52.910057 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:12:52.910026 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4eb9b6_b3e2_4768_96db_5908938e1871.slice/crio-2f1deca3dafc2364d2c5db7725813a615be7601c64c095d7c30af6c1828951d2 WatchSource:0}: Error finding container 2f1deca3dafc2364d2c5db7725813a615be7601c64c095d7c30af6c1828951d2: Status 404 returned error can't find the container with id 2f1deca3dafc2364d2c5db7725813a615be7601c64c095d7c30af6c1828951d2 Jul 26 18:12:53.257301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:53.257256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" event={"ID":"aa4eb9b6-b3e2-4768-96db-5908938e1871","Type":"ContainerStarted","Data":"6cfdf1b9bc5cdf1f797fcf11c6b58facd25872e54626d8f7a7a542dae28165f6"} Jul 26 18:12:53.257301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:53.257304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" event={"ID":"aa4eb9b6-b3e2-4768-96db-5908938e1871","Type":"ContainerStarted","Data":"2f1deca3dafc2364d2c5db7725813a615be7601c64c095d7c30af6c1828951d2"} Jul 26 18:12:53.278636 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:12:53.278588 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" podStartSLOduration=1.278572263 podStartE2EDuration="1.278572263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:12:53.275836664 +0000 UTC m=+1403.674865706" watchObservedRunningTime="2026-07-26 18:12:53.278572263 +0000 UTC m=+1403.677601312" Jul 26 18:13:02.757062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:02.756962 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:13:02.757062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:02.757024 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:13:02.769471 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:02.769441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:13:03.316255 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:03.316221 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:13:03.473890 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:03.473853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:13:05.794979 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.794947 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:13:05.798644 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.798623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.801380 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.801362 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-09f1ce9d-kserve-self-signed-certs\"" Jul 26 18:13:05.810311 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.810279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:13:05.866958 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.866926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.867130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.866966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.867130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.867006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.867130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.867052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtdz8\" (UniqueName: \"kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.867130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.867081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.867130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.867103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968215 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtdz8\" (UniqueName: \"kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968544 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968544 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968544 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968695 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968695 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.968850 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.968833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.970695 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.970664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.970990 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.970969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:05.976765 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:05.976746 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtdz8\" (UniqueName: \"kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8\") pod \"e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:06.108844 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:06.108810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:06.244594 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:06.244568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:13:06.246971 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:13:06.246932 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae9ea4f_54e1_4810_b76f_3598eb0519f5.slice/crio-ff07332b3eb8a051b479c5a1c8657183885b86b4e4e6929fd2fbdb08a070b9e7 WatchSource:0}: Error finding container ff07332b3eb8a051b479c5a1c8657183885b86b4e4e6929fd2fbdb08a070b9e7: Status 404 returned error can't find the container with id ff07332b3eb8a051b479c5a1c8657183885b86b4e4e6929fd2fbdb08a070b9e7 Jul 26 18:13:06.323730 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:06.323690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" event={"ID":"0ae9ea4f-54e1-4810-b76f-3598eb0519f5","Type":"ContainerStarted","Data":"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271"} Jul 26 18:13:06.323911 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:06.323742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" event={"ID":"0ae9ea4f-54e1-4810-b76f-3598eb0519f5","Type":"ContainerStarted","Data":"ff07332b3eb8a051b479c5a1c8657183885b86b4e4e6929fd2fbdb08a070b9e7"} Jul 26 18:13:06.339964 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:06.339906 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" podStartSLOduration=1.339885094 podStartE2EDuration="1.339885094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:13:06.339027485 +0000 UTC m=+1416.738056530" watchObservedRunningTime="2026-07-26 18:13:06.339885094 +0000 UTC m=+1416.738914138" Jul 26 18:13:16.109755 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:16.109714 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:16.110244 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:16.109783 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:16.122293 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:16.122266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:16.388782 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:16.388701 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:13:17.605027 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:17.604987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:13:44.438813 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:44.438714 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:13:46.013158 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:46.013127 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:13:48.573955 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:48.573912 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:13:59.458225 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.458160 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" podUID="aa4eb9b6-b3e2-4768-96db-5908938e1871" containerName="main" containerID="cri-o://6cfdf1b9bc5cdf1f797fcf11c6b58facd25872e54626d8f7a7a542dae28165f6" gracePeriod=45 Jul 26 18:13:59.603540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.603498 2575 generic.go:356] "Generic (PLEG): container finished" podID="aa4eb9b6-b3e2-4768-96db-5908938e1871" containerID="6cfdf1b9bc5cdf1f797fcf11c6b58facd25872e54626d8f7a7a542dae28165f6" exitCode=0 Jul 26 18:13:59.603691 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.603563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" event={"ID":"aa4eb9b6-b3e2-4768-96db-5908938e1871","Type":"ContainerDied","Data":"6cfdf1b9bc5cdf1f797fcf11c6b58facd25872e54626d8f7a7a542dae28165f6"} Jul 26 18:13:59.708208 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.708185 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:13:59.898284 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898251 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qq6w\" (UniqueName: \"kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898454 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898295 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898454 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898344 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898454 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898365 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898454 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir\") pod \"aa4eb9b6-b3e2-4768-96db-5908938e1871\" (UID: \"aa4eb9b6-b3e2-4768-96db-5908938e1871\") " Jul 26 18:13:59.898718 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898656 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:13:59.898779 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898728 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home" (OuterVolumeSpecName: "home") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:13:59.898858 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.898771 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:13:59.900607 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.900584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm" (OuterVolumeSpecName: "dshm") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:13:59.900607 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.900598 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:13:59.900769 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.900653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w" (OuterVolumeSpecName: "kube-api-access-5qq6w") pod "aa4eb9b6-b3e2-4768-96db-5908938e1871" (UID: "aa4eb9b6-b3e2-4768-96db-5908938e1871"). InnerVolumeSpecName "kube-api-access-5qq6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:13:59.999578 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999535 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq6w\" (UniqueName: \"kubernetes.io/projected/aa4eb9b6-b3e2-4768-96db-5908938e1871-kube-api-access-5qq6w\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:13:59.999578 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999570 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:13:59.999578 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999585 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:13:59.999855 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999598 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4eb9b6-b3e2-4768-96db-5908938e1871-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:13:59.999855 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999608 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:13:59.999855 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:13:59.999644 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4eb9b6-b3e2-4768-96db-5908938e1871-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:00.610109 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:00.610075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" event={"ID":"aa4eb9b6-b3e2-4768-96db-5908938e1871","Type":"ContainerDied","Data":"2f1deca3dafc2364d2c5db7725813a615be7601c64c095d7c30af6c1828951d2"} Jul 26 18:14:00.610521 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:00.610114 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz" Jul 26 18:14:00.610521 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:00.610124 2575 scope.go:117] "RemoveContainer" containerID="6cfdf1b9bc5cdf1f797fcf11c6b58facd25872e54626d8f7a7a542dae28165f6" Jul 26 18:14:00.636295 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:00.636268 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:14:00.640099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:00.640077 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-09f1ce9d-kserve-6f8f9466c5-wz8kz"] Jul 26 18:14:01.033373 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.033255 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" podUID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" containerName="main" containerID="cri-o://83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271" gracePeriod=45 Jul 26 18:14:01.287769 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.287706 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:14:01.412808 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.412764 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.412870 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtdz8\" (UniqueName: \"kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.412891 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.412924 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.412955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413081 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache\") pod \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\" (UID: \"0ae9ea4f-54e1-4810-b76f-3598eb0519f5\") " Jul 26 18:14:01.413231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413205 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home" (OuterVolumeSpecName: "home") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:14:01.413231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:14:01.413381 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413336 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:14:01.413443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413422 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.413443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413436 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.413548 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.413450 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.415064 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.415040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8" (OuterVolumeSpecName: "kube-api-access-wtdz8") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "kube-api-access-wtdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:14:01.415181 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.415060 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:14:01.415365 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.415349 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm" (OuterVolumeSpecName: "dshm") pod "0ae9ea4f-54e1-4810-b76f-3598eb0519f5" (UID: "0ae9ea4f-54e1-4810-b76f-3598eb0519f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:14:01.514574 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.514544 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtdz8\" (UniqueName: \"kubernetes.io/projected/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-kube-api-access-wtdz8\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.514574 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.514572 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.514574 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.514581 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae9ea4f-54e1-4810-b76f-3598eb0519f5-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:14:01.615859 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.615825 2575 generic.go:356] "Generic (PLEG): container finished" podID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" containerID="83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271" exitCode=0 Jul 26 18:14:01.616317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.615903 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" Jul 26 18:14:01.616317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.615938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" event={"ID":"0ae9ea4f-54e1-4810-b76f-3598eb0519f5","Type":"ContainerDied","Data":"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271"} Jul 26 18:14:01.616317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.615983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm" event={"ID":"0ae9ea4f-54e1-4810-b76f-3598eb0519f5","Type":"ContainerDied","Data":"ff07332b3eb8a051b479c5a1c8657183885b86b4e4e6929fd2fbdb08a070b9e7"} Jul 26 18:14:01.616317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.616013 2575 scope.go:117] "RemoveContainer" containerID="83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271" Jul 26 18:14:01.625243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.625224 2575 scope.go:117] "RemoveContainer" containerID="83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271" Jul 26 18:14:01.625525 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:14:01.625508 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271\": container with ID starting with 83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271 not found: ID does not exist" containerID="83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271" Jul 26 18:14:01.625577 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.625534 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271"} err="failed to get container status \"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271\": rpc error: code = NotFound desc = could not find container \"83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271\": container with ID starting with 83680dacbad470c633110e7771217cf36717239163d25660aa37606014ceb271 not found: ID does not exist" Jul 26 18:14:01.639307 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.639279 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:14:01.643643 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:01.643621 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-09f1ce9d-kserve-57467499-k88nm"] Jul 26 18:14:02.153404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:02.153371 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:14:02.209502 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:02.209468 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" path="/var/lib/kubelet/pods/0ae9ea4f-54e1-4810-b76f-3598eb0519f5/volumes" Jul 26 18:14:02.209922 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:02.209906 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4eb9b6-b3e2-4768-96db-5908938e1871" path="/var/lib/kubelet/pods/aa4eb9b6-b3e2-4768-96db-5908938e1871/volumes" Jul 26 18:14:11.577911 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:11.577874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:14:20.484065 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:20.484032 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:14:34.584204 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:34.584160 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:14:43.672568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:43.672533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:14:54.085392 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:14:54.085305 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:03.284094 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:03.284056 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:12.767075 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:12.767035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:26.269826 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:26.269768 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:35.474551 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:35.474513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:45.271427 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:45.271386 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:15:55.068005 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:15:55.067972 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:16:07.466592 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:16:07.466551 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:16:16.268766 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:16:16.268730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:16:28.564348 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:16:28.564265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:16:39.866742 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:16:39.866702 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:16:48.764601 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:16:48.764565 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:17:01.275308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:01.275266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:17:16.165921 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:16.165885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:17:22.665131 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665101 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:17:22.665515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665485 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" containerName="main" Jul 26 18:17:22.665515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665496 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" containerName="main" Jul 26 18:17:22.665598 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665524 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4eb9b6-b3e2-4768-96db-5908938e1871" containerName="main" Jul 26 18:17:22.665598 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665530 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4eb9b6-b3e2-4768-96db-5908938e1871" containerName="main" Jul 26 18:17:22.665598 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665591 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4eb9b6-b3e2-4768-96db-5908938e1871" containerName="main" Jul 26 18:17:22.665598 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.665598 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ae9ea4f-54e1-4810-b76f-3598eb0519f5" containerName="main" Jul 26 18:17:22.668838 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.668818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.672097 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.672076 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0608e5c-openshift-default-dockercfg-9djz7\"" Jul 26 18:17:22.696775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.696750 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:17:22.760523 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760523 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.760815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.761098 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.761098 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.760882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgtb\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.861843 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.861964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgtb\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862483 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862483 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.862650 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.862631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.864251 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.864228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.864358 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.864319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.873105 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.873080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.873733 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.873711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgtb\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb\") pod \"e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:22.979032 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:22.978943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:23.122663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.122600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:17:23.125181 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:17:23.125144 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d289f5_8f74_478f_96bb_491c807fe257.slice/crio-e1642ed6d124fcf9a175253004bdca04525497617d24a7fc820ea70e597f8ebc WatchSource:0}: Error finding container e1642ed6d124fcf9a175253004bdca04525497617d24a7fc820ea70e597f8ebc: Status 404 returned error can't find the container with id e1642ed6d124fcf9a175253004bdca04525497617d24a7fc820ea70e597f8ebc Jul 26 18:17:23.127278 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.127244 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:23.127374 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.127310 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:23.127374 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.127338 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:23.650425 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.650381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerStarted","Data":"68a7542d347b86047fbbbad3e9ad505fe7d8e7b4f91bd4a5dc186a02e9ca859e"} Jul 26 18:17:23.650636 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.650448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerStarted","Data":"e1642ed6d124fcf9a175253004bdca04525497617d24a7fc820ea70e597f8ebc"} Jul 26 18:17:23.674339 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.674279 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podStartSLOduration=1.674258106 podStartE2EDuration="1.674258106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:17:23.671479283 +0000 UTC m=+1674.070508350" watchObservedRunningTime="2026-07-26 18:17:23.674258106 +0000 UTC m=+1674.073287152" Jul 26 18:17:23.979566 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:23.979485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:24.984956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:24.984917 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:25.660209 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:25.660179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:25.661051 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:25.661027 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:32.770926 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:32.770895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:17:40.661588 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.661550 2575 patch_prober.go:28] interesting pod/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 18:17:40.661996 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.661617 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 18:17:40.735556 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.735519 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6d289f5-8f74-478f-96bb-491c807fe257" containerID="68a7542d347b86047fbbbad3e9ad505fe7d8e7b4f91bd4a5dc186a02e9ca859e" exitCode=0 Jul 26 18:17:40.735738 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.735594 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerDied","Data":"68a7542d347b86047fbbbad3e9ad505fe7d8e7b4f91bd4a5dc186a02e9ca859e"} Jul 26 18:17:40.736088 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.736071 2575 scope.go:117] "RemoveContainer" containerID="68a7542d347b86047fbbbad3e9ad505fe7d8e7b4f91bd4a5dc186a02e9ca859e" Jul 26 18:17:40.737121 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.737090 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:40.737205 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.737159 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:40.737205 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:40.737190 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:17:41.746274 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:41.746237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerStarted","Data":"62f74865b5ec45a34ef4a03b15af97dc5693e132b68bc93604a55f324ebf6fe5"} Jul 26 18:17:41.979781 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:41.979741 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:41.980020 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:41.979997 2575 patch_prober.go:28] interesting pod/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 18:17:41.980074 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:41.980056 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 18:17:42.980150 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:42.980109 2575 patch_prober.go:28] interesting pod/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 18:17:42.980540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:42.980178 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 18:17:43.979676 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:43.979641 2575 patch_prober.go:28] interesting pod/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 18:17:43.979898 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:43.979698 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 18:17:44.979932 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:44.979893 2575 patch_prober.go:28] interesting pod/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 26 18:17:44.980344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:44.979964 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 26 18:17:45.982883 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:45.982853 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:45.983332 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:45.983125 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:17:45.983877 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:17:45.983860 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:18:03.862056 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:03.861971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:18:05.732452 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:05.732419 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:18:05.732873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:05.732668 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" containerID="cri-o://62f74865b5ec45a34ef4a03b15af97dc5693e132b68bc93604a55f324ebf6fe5" gracePeriod=30 Jul 26 18:18:06.707059 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.707023 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:06.711365 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.711342 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.714027 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.714002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-47ecefc5-gw-openshift-default-dockercfg-jdzbn\"" Jul 26 18:18:06.723193 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.723168 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:06.793135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97l7\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.793524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.793389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894121 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894121 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894333 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894333 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894522 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894522 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97l7\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894867 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.894936 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.894916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.895330 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.895304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.896378 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.896351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.896672 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.896654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.904840 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.904816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97l7\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:06.905127 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:06.905105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token\") pod \"e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:07.025404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.025295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:07.177304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.177279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:07.180021 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:18:07.179977 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd4f4f4_39ba_4d9d_b42d_fbb7e275f789.slice/crio-a592f839b82e74c821f390050f00c2471dca15cea0acbe1aba0606732fbefb2d WatchSource:0}: Error finding container a592f839b82e74c821f390050f00c2471dca15cea0acbe1aba0606732fbefb2d: Status 404 returned error can't find the container with id a592f839b82e74c821f390050f00c2471dca15cea0acbe1aba0606732fbefb2d Jul 26 18:18:07.183124 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.183081 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:07.183227 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.183172 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:07.183227 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.183209 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:07.893158 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.893124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerStarted","Data":"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621"} Jul 26 18:18:07.893737 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.893204 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerStarted","Data":"a592f839b82e74c821f390050f00c2471dca15cea0acbe1aba0606732fbefb2d"} Jul 26 18:18:07.914162 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:07.914111 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podStartSLOduration=1.914095806 podStartE2EDuration="1.914095806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:18:07.913135583 +0000 UTC m=+1718.312164653" watchObservedRunningTime="2026-07-26 18:18:07.914095806 +0000 UTC m=+1718.313124851" Jul 26 18:18:08.026464 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:08.026430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:08.033719 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:08.033674 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:08.900130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:08.900098 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:08.901134 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:08.901114 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:10.914479 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:10.914443 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6d289f5-8f74-478f-96bb-491c807fe257" containerID="62f74865b5ec45a34ef4a03b15af97dc5693e132b68bc93604a55f324ebf6fe5" exitCode=0 Jul 26 18:18:10.914920 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:10.914508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerDied","Data":"62f74865b5ec45a34ef4a03b15af97dc5693e132b68bc93604a55f324ebf6fe5"} Jul 26 18:18:10.914920 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:10.914586 2575 scope.go:117] "RemoveContainer" containerID="68a7542d347b86047fbbbad3e9ad505fe7d8e7b4f91bd4a5dc186a02e9ca859e" Jul 26 18:18:10.984772 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:10.984743 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:18:11.037468 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037385 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfgtb\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037639 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037639 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037639 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037639 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037593 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037894 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037894 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037719 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037894 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037745 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.037894 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037773 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:11.037894 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037810 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token\") pod \"b6d289f5-8f74-478f-96bb-491c807fe257\" (UID: \"b6d289f5-8f74-478f-96bb-491c807fe257\") " Jul 26 18:18:11.038160 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.037984 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:18:11.038259 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038217 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:11.038328 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038286 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:11.038499 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038471 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.038585 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038498 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6d289f5-8f74-478f-96bb-491c807fe257-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.038585 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038513 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.038585 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038526 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.038920 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.038891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:11.040127 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.040101 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:11.040321 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.040302 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb" (OuterVolumeSpecName: "kube-api-access-nfgtb") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "kube-api-access-nfgtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:18:11.040388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.040366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:18:11.040508 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.040489 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6d289f5-8f74-478f-96bb-491c807fe257" (UID: "b6d289f5-8f74-478f-96bb-491c807fe257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:18:11.140103 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.140064 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.140103 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.140101 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6d289f5-8f74-478f-96bb-491c807fe257-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.140103 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.140111 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.140334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.140120 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfgtb\" (UniqueName: \"kubernetes.io/projected/b6d289f5-8f74-478f-96bb-491c807fe257-kube-api-access-nfgtb\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.140334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.140129 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6d289f5-8f74-478f-96bb-491c807fe257-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:11.921245 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.921213 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" Jul 26 18:18:11.921706 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.921255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df" event={"ID":"b6d289f5-8f74-478f-96bb-491c807fe257","Type":"ContainerDied","Data":"e1642ed6d124fcf9a175253004bdca04525497617d24a7fc820ea70e597f8ebc"} Jul 26 18:18:11.921706 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.921314 2575 scope.go:117] "RemoveContainer" containerID="62f74865b5ec45a34ef4a03b15af97dc5693e132b68bc93604a55f324ebf6fe5" Jul 26 18:18:11.944050 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.944020 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:18:11.948726 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:11.948700 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0608e5c-openshift-default-d57c5b56-lw5df"] Jul 26 18:18:12.214074 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:12.213991 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" path="/var/lib/kubelet/pods/b6d289f5-8f74-478f-96bb-491c807fe257/volumes" Jul 26 18:18:15.061867 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:15.061829 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:18:22.991149 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.991055 2575 generic.go:356] "Generic (PLEG): container finished" podID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerID="cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621" exitCode=0 Jul 26 18:18:22.991149 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.991131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerDied","Data":"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621"} Jul 26 18:18:22.991593 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.991534 2575 scope.go:117] "RemoveContainer" containerID="cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621" Jul 26 18:18:22.992554 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.992521 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:22.992612 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.992597 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:22.992684 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:22.992636 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:23.900189 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:23.900158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:23.997761 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:23.997723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerStarted","Data":"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4"} Jul 26 18:18:24.026117 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:24.026084 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:24.027074 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:24.027049 2575 patch_prober.go:28] interesting pod/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 26 18:18:24.027191 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:24.027096 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 26 18:18:25.026714 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:25.026675 2575 patch_prober.go:28] interesting pod/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 26 18:18:25.027204 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:25.026745 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 26 18:18:26.025888 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:26.025853 2575 patch_prober.go:28] interesting pod/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 26 18:18:26.026072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:26.025905 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 26 18:18:27.025991 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:27.025955 2575 patch_prober.go:28] interesting pod/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 26 18:18:27.026389 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:27.026020 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 26 18:18:28.030150 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:28.030122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:28.030604 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:28.030398 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:28.031168 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:28.031147 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:47.764433 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:47.764391 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:18:49.597924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:49.597888 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:49.598440 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:49.598166 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" containerID="cri-o://1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4" gracePeriod=30 Jul 26 18:18:50.347662 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.347625 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:18:50.348283 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.348264 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:18:50.348378 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.348287 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:18:50.348436 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.348417 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:18:50.348436 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.348430 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:18:50.351869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.351847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.355585 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.355559 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-10cd5747-openshift-default-dockercfg-55xdh\"" Jul 26 18:18:50.368961 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.368926 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:18:50.431411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431610 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431610 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431610 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.431943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.431816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7hzs\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533000 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.532964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533199 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533199 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533327 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533327 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7hzs\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533327 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533494 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533494 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533494 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533494 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533716 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533526 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533716 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.533889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.533823 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.534043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.534023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.535365 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.535344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.535925 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.535903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.541620 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.541590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.541728 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.541692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7hzs\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs\") pod \"e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.661917 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.661873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:50.805729 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.805699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:18:50.807232 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:18:50.807205 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e1d9a7_d217_483c_be83_09b93c9256e7.slice/crio-065658e0d23e2537b482811eca9fb5c08dc3f61502adaae08f7134f1a65fdb69 WatchSource:0}: Error finding container 065658e0d23e2537b482811eca9fb5c08dc3f61502adaae08f7134f1a65fdb69: Status 404 returned error can't find the container with id 065658e0d23e2537b482811eca9fb5c08dc3f61502adaae08f7134f1a65fdb69 Jul 26 18:18:50.809520 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.809484 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:50.809631 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.809558 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:50.809631 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:50.809619 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:18:51.142784 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:51.142749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerStarted","Data":"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea"} Jul 26 18:18:51.142958 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:51.142836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerStarted","Data":"065658e0d23e2537b482811eca9fb5c08dc3f61502adaae08f7134f1a65fdb69"} Jul 26 18:18:51.167945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:51.167881 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podStartSLOduration=1.167858115 podStartE2EDuration="1.167858115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:18:51.166981055 +0000 UTC m=+1761.566010104" watchObservedRunningTime="2026-07-26 18:18:51.167858115 +0000 UTC m=+1761.566887160" Jul 26 18:18:51.662367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:51.662329 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:51.667773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:51.667742 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:52.148636 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:52.148605 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:52.149575 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:52.149556 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:18:54.861163 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.861134 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:54.976745 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976653 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976745 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976687 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976757 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976841 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d97l7\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976884 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.976978 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.976967 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.977304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket\") pod \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\" (UID: \"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789\") " Jul 26 18:18:54.977304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977066 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:54.977304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977117 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:54.977304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977219 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:54.977533 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977366 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:54.977533 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977376 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:54.977533 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977386 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:54.977658 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977538 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data" (OuterVolumeSpecName: "istio-data") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:54.977658 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.977549 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:18:54.979446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.979398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:18:54.979689 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.979652 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:18:54.979785 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.979730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7" (OuterVolumeSpecName: "kube-api-access-d97l7") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "kube-api-access-d97l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:18:54.979785 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:54.979755 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token" (OuterVolumeSpecName: "istio-token") pod "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" (UID: "afd4f4f4-39ba-4d9d-b42d-fbb7e275f789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:18:55.077934 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077891 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97l7\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-kube-api-access-d97l7\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.077934 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077927 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.077934 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077939 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.077934 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077946 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.078209 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077954 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.078209 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.077961 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:18:55.172215 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.172180 2575 generic.go:356] "Generic (PLEG): container finished" podID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerID="1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4" exitCode=0 Jul 26 18:18:55.172410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.172230 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" Jul 26 18:18:55.172410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.172267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerDied","Data":"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4"} Jul 26 18:18:55.172410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.172332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv" event={"ID":"afd4f4f4-39ba-4d9d-b42d-fbb7e275f789","Type":"ContainerDied","Data":"a592f839b82e74c821f390050f00c2471dca15cea0acbe1aba0606732fbefb2d"} Jul 26 18:18:55.172410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.172366 2575 scope.go:117] "RemoveContainer" containerID="1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4" Jul 26 18:18:55.181892 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.181876 2575 scope.go:117] "RemoveContainer" containerID="cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621" Jul 26 18:18:55.189730 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.189713 2575 scope.go:117] "RemoveContainer" containerID="1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4" Jul 26 18:18:55.190001 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:18:55.189983 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4\": container with ID starting with 1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4 not found: ID does not exist" containerID="1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4" Jul 26 18:18:55.190086 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.190014 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4"} err="failed to get container status \"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4\": rpc error: code = NotFound desc = could not find container \"1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4\": container with ID starting with 1d9ce5f8580cf8e6502b3d414a3f8d17b86702977dfaea2bf2d064a32f2195a4 not found: ID does not exist" Jul 26 18:18:55.190086 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.190038 2575 scope.go:117] "RemoveContainer" containerID="cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621" Jul 26 18:18:55.190307 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:18:55.190285 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621\": container with ID starting with cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621 not found: ID does not exist" containerID="cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621" Jul 26 18:18:55.190353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.190313 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621"} err="failed to get container status \"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621\": rpc error: code = NotFound desc = could not find container \"cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621\": container with ID starting with cde92799dcb643e5fb01c291eb4c03197dd6bdb6686f86201c0632f17ad07621 not found: ID does not exist" Jul 26 18:18:55.193967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.193944 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:55.200019 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:55.199994 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-47ecefc5-gw-openshift-default-84684d6596-hs6jv"] Jul 26 18:18:56.209041 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:56.209005 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" path="/var/lib/kubelet/pods/afd4f4f4-39ba-4d9d-b42d-fbb7e275f789/volumes" Jul 26 18:18:58.567445 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:18:58.567412 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:19:07.150247 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.150201 2575 patch_prober.go:28] interesting pod/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 26 18:19:07.150725 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.150275 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 26 18:19:07.246182 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.246149 2575 generic.go:356] "Generic (PLEG): container finished" podID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerID="1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea" exitCode=0 Jul 26 18:19:07.246382 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.246210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerDied","Data":"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea"} Jul 26 18:19:07.246621 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.246602 2575 scope.go:117] "RemoveContainer" containerID="1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea" Jul 26 18:19:07.247658 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.247627 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:07.247739 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.247696 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:07.247739 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:07.247728 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:08.255038 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:08.255005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerStarted","Data":"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c"} Jul 26 18:19:08.662310 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:08.662262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:08.662549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:08.662525 2575 patch_prober.go:28] interesting pod/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 26 18:19:08.662625 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:08.662579 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 26 18:19:09.662385 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:09.662345 2575 patch_prober.go:28] interesting pod/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 26 18:19:09.662883 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:09.662407 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 26 18:19:10.663312 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:10.663262 2575 patch_prober.go:28] interesting pod/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 26 18:19:10.663861 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:10.663318 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 26 18:19:11.662718 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:11.662672 2575 patch_prober.go:28] interesting pod/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 26 18:19:11.662968 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:11.662747 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 26 18:19:12.666504 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:12.666471 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:12.666913 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:12.666820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:12.667489 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:12.667470 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:31.672852 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:31.672815 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:19:34.628481 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:34.628447 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:19:34.628919 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:34.628666 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" containerID="cri-o://7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c" gracePeriod=30 Jul 26 18:19:35.418918 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.418885 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:19:35.419375 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419359 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:19:35.419438 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419377 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:19:35.419438 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419395 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:19:35.419438 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419401 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d289f5-8f74-478f-96bb-491c807fe257" containerName="istio-proxy" Jul 26 18:19:35.419535 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419461 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:19:35.419535 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.419473 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:19:35.422650 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.422627 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.426256 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.426232 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-535bf7ee-openshift-default-dockercfg-2rll4\"" Jul 26 18:19:35.440829 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.440800 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:19:35.597478 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597434 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597478 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.597754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.598053 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjc5\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.598053 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.597882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.698852 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.698749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.698852 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.698846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.698874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.698909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjc5\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699307 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699856 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699856 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.699856 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.699826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.701137 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.701116 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.701651 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.701629 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.707619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.707592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjc5\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.708304 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.708285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token\") pod \"e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.733466 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.733436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:35.882408 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.882379 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:19:35.884265 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:19:35.884232 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08892a4_6206_4615_b0ac_a40cc5ed325e.slice/crio-672336ba122a524929ba273cfc997a9c88767755bbd80936717030b7b1f9262a WatchSource:0}: Error finding container 672336ba122a524929ba273cfc997a9c88767755bbd80936717030b7b1f9262a: Status 404 returned error can't find the container with id 672336ba122a524929ba273cfc997a9c88767755bbd80936717030b7b1f9262a Jul 26 18:19:35.886398 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.886364 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:35.886467 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.886429 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:35.886467 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:35.886459 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:36.408212 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:36.408178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerStarted","Data":"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df"} Jul 26 18:19:36.408435 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:36.408239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerStarted","Data":"672336ba122a524929ba273cfc997a9c88767755bbd80936717030b7b1f9262a"} Jul 26 18:19:36.431084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:36.431036 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podStartSLOduration=1.431020649 podStartE2EDuration="1.431020649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:19:36.42917752 +0000 UTC m=+1806.828206576" watchObservedRunningTime="2026-07-26 18:19:36.431020649 +0000 UTC m=+1806.830049738" Jul 26 18:19:36.734276 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:36.734177 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:36.739320 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:36.739292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:37.413313 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:37.413281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:37.414220 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:37.414199 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:39.888164 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.888094 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:39.936760 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936726 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7hzs\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.936975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.936975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.936975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936917 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.936975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936943 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.936975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936973 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.937239 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.936997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.937239 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937047 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.937239 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket\") pod \"f7e1d9a7-d217-483c-be83-09b93c9256e7\" (UID: \"f7e1d9a7-d217-483c-be83-09b93c9256e7\") " Jul 26 18:19:39.937897 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:19:39.937897 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:19:39.937897 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937869 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:19:39.938161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.937988 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:19:39.938161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.938014 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:19:39.939123 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.939100 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs" (OuterVolumeSpecName: "kube-api-access-l7hzs") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "kube-api-access-l7hzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:19:39.939267 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.939249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:19:39.939373 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.939350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:19:39.939484 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:39.939409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7e1d9a7-d217-483c-be83-09b93c9256e7" (UID: "f7e1d9a7-d217-483c-be83-09b93c9256e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:19:40.038062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038032 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038057 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038066 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038062 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038074 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038083 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e1d9a7-d217-483c-be83-09b93c9256e7-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038091 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038100 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038108 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e1d9a7-d217-483c-be83-09b93c9256e7-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.038334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.038116 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7hzs\" (UniqueName: \"kubernetes.io/projected/f7e1d9a7-d217-483c-be83-09b93c9256e7-kube-api-access-l7hzs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:19:40.431309 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.431258 2575 generic.go:356] "Generic (PLEG): container finished" podID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerID="7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c" exitCode=0 Jul 26 18:19:40.431309 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.431288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerDied","Data":"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c"} Jul 26 18:19:40.431529 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.431335 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" Jul 26 18:19:40.431529 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.431386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm" event={"ID":"f7e1d9a7-d217-483c-be83-09b93c9256e7","Type":"ContainerDied","Data":"065658e0d23e2537b482811eca9fb5c08dc3f61502adaae08f7134f1a65fdb69"} Jul 26 18:19:40.431529 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.431411 2575 scope.go:117] "RemoveContainer" containerID="7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c" Jul 26 18:19:40.440729 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.440709 2575 scope.go:117] "RemoveContainer" containerID="1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea" Jul 26 18:19:40.448856 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.448830 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:19:40.449430 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.449409 2575 scope.go:117] "RemoveContainer" containerID="7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c" Jul 26 18:19:40.449748 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:19:40.449728 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c\": container with ID starting with 7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c not found: ID does not exist" containerID="7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c" Jul 26 18:19:40.449814 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.449757 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c"} err="failed to get container status \"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c\": rpc error: code = NotFound desc = could not find container \"7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c\": container with ID starting with 7df675fef69054141743112e813fe516afc922eedd123cbe6f606e7389c4ac3c not found: ID does not exist" Jul 26 18:19:40.449814 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.449776 2575 scope.go:117] "RemoveContainer" containerID="1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea" Jul 26 18:19:40.450080 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:19:40.450046 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea\": container with ID starting with 1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea not found: ID does not exist" containerID="1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea" Jul 26 18:19:40.450080 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.450070 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea"} err="failed to get container status \"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea\": rpc error: code = NotFound desc = could not find container \"1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea\": container with ID starting with 1bf0d9eaaa0ed955f9d4a81b3ad324491f26a39d66875840cc12e6f04ac655ea not found: ID does not exist" Jul 26 18:19:40.452876 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:40.452855 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-10cd5747-openshift-default-5fcc9686c8-g7mfm"] Jul 26 18:19:42.209005 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:42.208971 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" path="/var/lib/kubelet/pods/f7e1d9a7-d217-483c-be83-09b93c9256e7/volumes" Jul 26 18:19:45.374876 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:45.374834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:19:53.190623 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.190165 2575 patch_prober.go:28] interesting pod/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:60176->10.134.0.42:15021: read: connection reset by peer" start-of-body= Jul 26 18:19:53.190623 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.190237 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:60176->10.134.0.42:15021: read: connection reset by peer" Jul 26 18:19:53.502375 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.502289 2575 generic.go:356] "Generic (PLEG): container finished" podID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerID="9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df" exitCode=0 Jul 26 18:19:53.502524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.502367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerDied","Data":"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df"} Jul 26 18:19:53.502816 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.502801 2575 scope.go:117] "RemoveContainer" containerID="9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df" Jul 26 18:19:53.503774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.503748 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:53.503853 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.503839 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:53.503891 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:53.503878 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:19:54.508815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:54.508755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerStarted","Data":"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e"} Jul 26 18:19:54.734206 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:54.734167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:54.734409 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:54.734294 2575 patch_prober.go:28] interesting pod/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 26 18:19:54.734409 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:54.734340 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 26 18:19:55.734557 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:55.734510 2575 patch_prober.go:28] interesting pod/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 26 18:19:55.735061 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:55.734580 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 26 18:19:56.734136 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:56.734095 2575 patch_prober.go:28] interesting pod/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 26 18:19:56.734897 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:56.734434 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 26 18:19:57.734747 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:57.734687 2575 patch_prober.go:28] interesting pod/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 26 18:19:57.735208 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:57.734776 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 26 18:19:58.738040 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:58.738006 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:58.738446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:58.738286 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:19:58.739000 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:19:58.738981 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:20:16.068876 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:16.068841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:20:17.838830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:17.838768 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:20:17.839263 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:17.839061 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" containerID="cri-o://cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e" gracePeriod=30 Jul 26 18:20:23.099539 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.099498 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:20:23.179268 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179291 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179325 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179363 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179393 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179421 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179720 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179453 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179720 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179534 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179720 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:20:23.179720 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179586 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpjc5\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5\") pod \"b08892a4-6206-4615-b0ac-a40cc5ed325e\" (UID: \"b08892a4-6206-4615-b0ac-a40cc5ed325e\") " Jul 26 18:20:23.179962 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179812 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:20:23.179962 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:20:23.179962 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179883 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:20:23.180106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.179992 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.180106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.180013 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.180106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.180028 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08892a4-6206-4615-b0ac-a40cc5ed325e-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.180106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.180042 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.180842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.180814 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data" (OuterVolumeSpecName: "istio-data") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:20:23.181608 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.181575 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:20:23.181711 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.181626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:20:23.181711 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.181683 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:20:23.181940 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.181921 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5" (OuterVolumeSpecName: "kube-api-access-wpjc5") pod "b08892a4-6206-4615-b0ac-a40cc5ed325e" (UID: "b08892a4-6206-4615-b0ac-a40cc5ed325e"). InnerVolumeSpecName "kube-api-access-wpjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:20:23.280905 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.280871 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.280905 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.280901 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.280905 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.280911 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.281150 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.280919 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08892a4-6206-4615-b0ac-a40cc5ed325e-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.281150 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.280929 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjc5\" (UniqueName: \"kubernetes.io/projected/b08892a4-6206-4615-b0ac-a40cc5ed325e-kube-api-access-wpjc5\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:20:23.655442 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.655405 2575 generic.go:356] "Generic (PLEG): container finished" podID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerID="cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e" exitCode=0 Jul 26 18:20:23.655622 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.655463 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" Jul 26 18:20:23.655622 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.655462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerDied","Data":"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e"} Jul 26 18:20:23.655622 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.655583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm" event={"ID":"b08892a4-6206-4615-b0ac-a40cc5ed325e","Type":"ContainerDied","Data":"672336ba122a524929ba273cfc997a9c88767755bbd80936717030b7b1f9262a"} Jul 26 18:20:23.655622 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.655604 2575 scope.go:117] "RemoveContainer" containerID="cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e" Jul 26 18:20:23.665086 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.665067 2575 scope.go:117] "RemoveContainer" containerID="9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df" Jul 26 18:20:23.673010 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.672991 2575 scope.go:117] "RemoveContainer" containerID="cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e" Jul 26 18:20:23.673241 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:20:23.673224 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e\": container with ID starting with cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e not found: ID does not exist" containerID="cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e" Jul 26 18:20:23.673285 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.673250 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e"} err="failed to get container status \"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e\": rpc error: code = NotFound desc = could not find container \"cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e\": container with ID starting with cd87e2b37a2a2350fdf43a28829b817e76aa3a927b653fdbc79de91a8591605e not found: ID does not exist" Jul 26 18:20:23.673285 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.673267 2575 scope.go:117] "RemoveContainer" containerID="9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df" Jul 26 18:20:23.673495 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:20:23.673477 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df\": container with ID starting with 9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df not found: ID does not exist" containerID="9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df" Jul 26 18:20:23.673542 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.673500 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df"} err="failed to get container status \"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df\": rpc error: code = NotFound desc = could not find container \"9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df\": container with ID starting with 9c86f27251117dedfc67aa952d366161c6d682830186d6fee862a0fbd740f8df not found: ID does not exist" Jul 26 18:20:23.678858 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.678833 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:20:23.682545 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:23.682522 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-535bf7ee-openshift-default-567fccf4d4-j6vqm"] Jul 26 18:20:24.209058 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:20:24.209025 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" path="/var/lib/kubelet/pods/b08892a4-6206-4615-b0ac-a40cc5ed325e/volumes" Jul 26 18:21:42.072344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:42.072294 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:21:52.669726 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:52.669690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:21:54.676710 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.676674 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:21:54.677130 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677116 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677132 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677145 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677151 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677158 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677163 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677169 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677174 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677198 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677203 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd4f4f4-39ba-4d9d-b42d-fbb7e275f789" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677256 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677266 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08892a4-6206-4615-b0ac-a40cc5ed325e" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677272 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.677390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.677279 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e1d9a7-d217-483c-be83-09b93c9256e7" containerName="istio-proxy" Jul 26 18:21:54.680443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.680427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.683148 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.683126 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d700fe05-openshift-default-dockercfg-9nwsr\"" Jul 26 18:21:54.701619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.701592 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:21:54.763081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwcw\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763646 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763646 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.763646 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.763479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864534 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864534 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864534 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864534 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwcw\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864772 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864772 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864772 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.864772 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.865002 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.864875 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.865366 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.865255 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.865681 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.865653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.865877 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.865693 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.866015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.865887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.867653 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.867627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.867839 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.867817 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.872596 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.872574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.872679 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.872616 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwcw\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw\") pod \"e2e-mt-d700fe05-openshift-default-b68c97596-nn85s\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:54.990669 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:54.990586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:55.130063 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:55.130037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:21:55.131513 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:21:55.131478 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f50a34_87b6_4be0_b572_c8457d888010.slice/crio-d055295d7f4892e32661c8164f3d8077c965fbda874c6193c2490f82bf0444f3 WatchSource:0}: Error finding container d055295d7f4892e32661c8164f3d8077c965fbda874c6193c2490f82bf0444f3: Status 404 returned error can't find the container with id d055295d7f4892e32661c8164f3d8077c965fbda874c6193c2490f82bf0444f3 Jul 26 18:21:55.133831 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:55.133768 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:21:55.133939 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:55.133884 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:21:55.133939 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:55.133929 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:21:56.138527 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:56.138491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerStarted","Data":"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77"} Jul 26 18:21:56.138945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:56.138540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerStarted","Data":"d055295d7f4892e32661c8164f3d8077c965fbda874c6193c2490f82bf0444f3"} Jul 26 18:21:56.160167 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:56.160114 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podStartSLOduration=2.160098064 podStartE2EDuration="2.160098064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:21:56.158859393 +0000 UTC m=+1946.557888439" watchObservedRunningTime="2026-07-26 18:21:56.160098064 +0000 UTC m=+1946.559127108" Jul 26 18:21:56.991125 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:56.991084 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:56.996276 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:56.996241 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:57.143828 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:57.143773 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:21:57.144887 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:21:57.144868 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:01.960433 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:01.960398 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:22:10.216360 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.216327 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4f50a34-87b6-4be0-b572-c8457d888010" containerID="ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77" exitCode=0 Jul 26 18:22:10.216360 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.216363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerDied","Data":"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77"} Jul 26 18:22:10.216822 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.216703 2575 scope.go:117] "RemoveContainer" containerID="ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77" Jul 26 18:22:10.217650 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.217622 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:10.217706 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.217683 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:10.217752 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:10.217714 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:11.223941 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:11.223907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerStarted","Data":"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2"} Jul 26 18:22:11.991713 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:11.991665 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:11.992943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:11.992913 2575 patch_prober.go:28] interesting pod/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 26 18:22:11.993072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:11.992966 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 26 18:22:12.144600 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:12.144551 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:12.991742 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:12.991707 2575 patch_prober.go:28] interesting pod/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 26 18:22:12.992140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:12.991762 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 26 18:22:13.992037 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:13.991994 2575 patch_prober.go:28] interesting pod/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 26 18:22:13.992423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:13.992053 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 26 18:22:14.991871 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:14.991837 2575 patch_prober.go:28] interesting pod/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 26 18:22:14.992051 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:14.991892 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 26 18:22:15.994838 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:15.994783 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:15.995663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:15.995645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:23.071121 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:23.071044 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:22:31.310232 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:31.310190 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:22:31.310732 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:31.310432 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" containerID="cri-o://61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2" gracePeriod=30 Jul 26 18:22:36.564904 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.564877 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:36.691664 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691629 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.691861 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.691861 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691722 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.691861 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691753 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.691861 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gwcw\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.692073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691883 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.692073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.692073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691934 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.692073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.691969 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs\") pod \"b4f50a34-87b6-4be0-b572-c8457d888010\" (UID: \"b4f50a34-87b6-4be0-b572-c8457d888010\") " Jul 26 18:22:36.692295 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692075 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:22:36.692295 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:22:36.692295 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692268 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:22:36.692448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692410 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.692448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692432 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.692540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692448 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.692540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:22:36.692624 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.692563 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:22:36.693988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.693963 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:22:36.694158 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.694140 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:22:36.694401 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.694381 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:22:36.694476 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.694427 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw" (OuterVolumeSpecName: "kube-api-access-4gwcw") pod "b4f50a34-87b6-4be0-b572-c8457d888010" (UID: "b4f50a34-87b6-4be0-b572-c8457d888010"). InnerVolumeSpecName "kube-api-access-4gwcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:22:36.793654 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793614 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.793654 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793647 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.793654 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793656 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f50a34-87b6-4be0-b572-c8457d888010-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.793654 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793664 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gwcw\" (UniqueName: \"kubernetes.io/projected/b4f50a34-87b6-4be0-b572-c8457d888010-kube-api-access-4gwcw\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.793962 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793674 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f50a34-87b6-4be0-b572-c8457d888010-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:36.793962 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:36.793683 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f50a34-87b6-4be0-b572-c8457d888010-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:22:37.366173 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.366138 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4f50a34-87b6-4be0-b572-c8457d888010" containerID="61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2" exitCode=0 Jul 26 18:22:37.366367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.366187 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" Jul 26 18:22:37.366367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.366231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerDied","Data":"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2"} Jul 26 18:22:37.366367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.366290 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s" event={"ID":"b4f50a34-87b6-4be0-b572-c8457d888010","Type":"ContainerDied","Data":"d055295d7f4892e32661c8164f3d8077c965fbda874c6193c2490f82bf0444f3"} Jul 26 18:22:37.366367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.366319 2575 scope.go:117] "RemoveContainer" containerID="61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2" Jul 26 18:22:37.376054 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.376036 2575 scope.go:117] "RemoveContainer" containerID="ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77" Jul 26 18:22:37.384005 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.383984 2575 scope.go:117] "RemoveContainer" containerID="61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2" Jul 26 18:22:37.384276 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:22:37.384256 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2\": container with ID starting with 61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2 not found: ID does not exist" containerID="61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2" Jul 26 18:22:37.384329 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.384285 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2"} err="failed to get container status \"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2\": rpc error: code = NotFound desc = could not find container \"61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2\": container with ID starting with 61a90eb225f7e05cf4fd8263ef4555f0f75eb0673f2c0705ed9bfc9ea03db4d2 not found: ID does not exist" Jul 26 18:22:37.384329 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.384304 2575 scope.go:117] "RemoveContainer" containerID="ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77" Jul 26 18:22:37.384553 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:22:37.384534 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77\": container with ID starting with ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77 not found: ID does not exist" containerID="ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77" Jul 26 18:22:37.384609 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.384562 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77"} err="failed to get container status \"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77\": rpc error: code = NotFound desc = could not find container \"ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77\": container with ID starting with ddc418a5ae049740184fa286f74e6e2471b1c6bf0a8f7cdef34e2f20a47a6d77 not found: ID does not exist" Jul 26 18:22:37.388547 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.388524 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:22:37.393574 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:37.393553 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d700fe05-openshift-default-b68c97596-nn85s"] Jul 26 18:22:38.208436 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:38.208400 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" path="/var/lib/kubelet/pods/b4f50a34-87b6-4be0-b572-c8457d888010/volumes" Jul 26 18:22:51.118557 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.118526 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:22:51.119031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.118983 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.119031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.118996 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.119031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.119004 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.119031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.119010 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.119161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.119069 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.119161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.119079 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f50a34-87b6-4be0-b572-c8457d888010" containerName="istio-proxy" Jul 26 18:22:51.124335 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.124311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.127879 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.127859 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-614abb29-openshift-default-dockercfg-6ls6j\"" Jul 26 18:22:51.145014 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.144985 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:22:51.228214 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228400 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228400 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228400 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhrd\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228525 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228525 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228525 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228525 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.228672 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.228526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329643 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329643 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhrd\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.329956 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.329918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.330364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.330081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.330364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.330110 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.330364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.330166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.330638 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.330613 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.330736 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.330725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.332159 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.332138 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.332330 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.332314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.338891 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.338867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.339021 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.338966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhrd\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd\") pod \"e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.434936 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.434841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:51.571101 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.571075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:22:51.573057 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:22:51.573034 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc72c98_90b6_4176_ba25_1e0f0ac2c2f5.slice/crio-a0b7b16d8dd50b726722af1ed4e45c904ecca710b53376beccd3ce1fdcc6713e WatchSource:0}: Error finding container a0b7b16d8dd50b726722af1ed4e45c904ecca710b53376beccd3ce1fdcc6713e: Status 404 returned error can't find the container with id a0b7b16d8dd50b726722af1ed4e45c904ecca710b53376beccd3ce1fdcc6713e Jul 26 18:22:51.575691 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.575655 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:51.575759 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.575742 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:51.575832 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:51.575810 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:22:52.448877 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:52.448841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerStarted","Data":"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d"} Jul 26 18:22:52.449274 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:52.448895 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerStarted","Data":"a0b7b16d8dd50b726722af1ed4e45c904ecca710b53376beccd3ce1fdcc6713e"} Jul 26 18:22:52.474193 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:52.474141 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podStartSLOduration=1.474124709 podStartE2EDuration="1.474124709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:22:52.472001648 +0000 UTC m=+2002.871030693" watchObservedRunningTime="2026-07-26 18:22:52.474124709 +0000 UTC m=+2002.873153732" Jul 26 18:22:53.435361 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:53.435323 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:53.440608 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:53.440579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:53.454432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:53.454407 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:22:53.455325 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:22:53.455305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:23:01.369453 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:01.369421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:23:06.928640 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.928564 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:23:06.932210 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.932190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:06.934974 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.934950 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-614abb29\"/\"kube-root-ca.crt\"" Jul 26 18:23:06.936135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.936112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-614abb29\"/\"openshift-service-ca.crt\"" Jul 26 18:23:06.936222 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.936112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-614abb29\"/\"default-dockercfg-nntpc\"" Jul 26 18:23:06.936335 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.936315 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-614abb29\"/\"e2e-lifecycle-model-614abb29-kserve-self-signed-certs\"" Jul 26 18:23:06.948152 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:06.948123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:23:07.088054 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5ff\" (UniqueName: \"kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.088248 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.088248 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.088248 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.092813 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.092813 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.088695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189586 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5ff\" (UniqueName: \"kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189756 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189756 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189756 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189756 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.189756 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.189752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.190305 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.190274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.190489 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.190302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.190614 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.190470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.192863 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.192839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.194779 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.194752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.204500 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.204473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5ff\" (UniqueName: \"kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff\") pod \"e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.241923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.241889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:07.408879 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.408721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:23:07.411870 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:23:07.411836 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeefc72af_cfce_434c_a442_d632894d1448.slice/crio-ee2a9758c1c5850a950dcc37d4c7dc85605d926f373d47e200181ee1993bc28a WatchSource:0}: Error finding container ee2a9758c1c5850a950dcc37d4c7dc85605d926f373d47e200181ee1993bc28a: Status 404 returned error can't find the container with id ee2a9758c1c5850a950dcc37d4c7dc85605d926f373d47e200181ee1993bc28a Jul 26 18:23:07.527568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.527524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" event={"ID":"eefc72af-cfce-434c-a442-d632894d1448","Type":"ContainerStarted","Data":"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb"} Jul 26 18:23:07.527568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.527581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" event={"ID":"eefc72af-cfce-434c-a442-d632894d1448","Type":"ContainerStarted","Data":"ee2a9758c1c5850a950dcc37d4c7dc85605d926f373d47e200181ee1993bc28a"} Jul 26 18:23:07.551994 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:07.551939 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" podStartSLOduration=1.55191919 podStartE2EDuration="1.55191919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:23:07.549107917 +0000 UTC m=+2017.948136963" watchObservedRunningTime="2026-07-26 18:23:07.55191919 +0000 UTC m=+2017.950948236" Jul 26 18:23:09.448994 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.447909 2575 patch_prober.go:28] interesting pod/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:41152->10.134.0.44:15021: read: connection reset by peer" start-of-body= Jul 26 18:23:09.448994 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.447976 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:41152->10.134.0.44:15021: read: connection reset by peer" Jul 26 18:23:09.540081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.540048 2575 generic.go:356] "Generic (PLEG): container finished" podID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerID="bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d" exitCode=0 Jul 26 18:23:09.540271 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.540079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerDied","Data":"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d"} Jul 26 18:23:09.540593 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.540577 2575 scope.go:117] "RemoveContainer" containerID="bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d" Jul 26 18:23:09.541655 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.541628 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:23:09.541717 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.541695 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:23:09.541778 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.541726 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:23:09.798017 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:09.797932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:23:10.553842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:10.553785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerStarted","Data":"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd"} Jul 26 18:23:11.435540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:11.435294 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:23:11.435540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:11.435457 2575 patch_prober.go:28] interesting pod/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 18:23:11.435540 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:11.435505 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 18:23:12.435327 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:12.435290 2575 patch_prober.go:28] interesting pod/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 18:23:12.435842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:12.435359 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 18:23:13.435838 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:13.435774 2575 patch_prober.go:28] interesting pod/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 26 18:23:13.436340 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:13.435868 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 26 18:23:14.438328 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:14.438296 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:23:14.438837 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:14.438696 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:23:14.439428 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:14.439410 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:23:17.242682 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:17.242647 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:17.243153 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:17.242697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:17.255385 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:17.255352 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:17.603222 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:17.603194 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:23:24.067519 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:24.067481 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:23:42.063859 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:42.063827 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:23:50.672635 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:50.672603 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:23:56.286003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:23:56.285901 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:24:11.305503 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.305456 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" podUID="eefc72af-cfce-434c-a442-d632894d1448" containerName="main" containerID="cri-o://5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb" gracePeriod=45 Jul 26 18:24:11.562246 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.562183 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:24:11.624950 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.624917 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625133 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.624960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625133 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.624997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr5ff\" (UniqueName: \"kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625133 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625133 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625071 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625351 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625139 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir\") pod \"eefc72af-cfce-434c-a442-d632894d1448\" (UID: \"eefc72af-cfce-434c-a442-d632894d1448\") " Jul 26 18:24:11.625351 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625196 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache" (OuterVolumeSpecName: "model-cache") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:11.625462 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625362 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home" (OuterVolumeSpecName: "home") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:11.625519 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625484 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:11.625580 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625526 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.625580 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.625540 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.627392 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.627361 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm" (OuterVolumeSpecName: "dshm") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:11.627513 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.627386 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff" (OuterVolumeSpecName: "kube-api-access-nr5ff") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "kube-api-access-nr5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:24:11.627513 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.627426 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eefc72af-cfce-434c-a442-d632894d1448" (UID: "eefc72af-cfce-434c-a442-d632894d1448"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:24:11.726432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.726401 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.726432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.726430 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eefc72af-cfce-434c-a442-d632894d1448-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.726432 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.726445 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eefc72af-cfce-434c-a442-d632894d1448-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.726674 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.726457 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr5ff\" (UniqueName: \"kubernetes.io/projected/eefc72af-cfce-434c-a442-d632894d1448-kube-api-access-nr5ff\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:11.870731 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.870696 2575 generic.go:356] "Generic (PLEG): container finished" podID="eefc72af-cfce-434c-a442-d632894d1448" containerID="5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb" exitCode=0 Jul 26 18:24:11.870945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.870738 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" Jul 26 18:24:11.870945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.870780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" event={"ID":"eefc72af-cfce-434c-a442-d632894d1448","Type":"ContainerDied","Data":"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb"} Jul 26 18:24:11.870945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.870839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx" event={"ID":"eefc72af-cfce-434c-a442-d632894d1448","Type":"ContainerDied","Data":"ee2a9758c1c5850a950dcc37d4c7dc85605d926f373d47e200181ee1993bc28a"} Jul 26 18:24:11.870945 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.870861 2575 scope.go:117] "RemoveContainer" containerID="5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb" Jul 26 18:24:11.880646 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.880629 2575 scope.go:117] "RemoveContainer" containerID="5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb" Jul 26 18:24:11.880948 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:24:11.880929 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb\": container with ID starting with 5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb not found: ID does not exist" containerID="5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb" Jul 26 18:24:11.881014 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.880957 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb"} err="failed to get container status \"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb\": rpc error: code = NotFound desc = could not find container \"5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb\": container with ID starting with 5082f7e07793c628c93ba9d4f73c6592c26c0c11b625af8f8a1b45d9cd9d3beb not found: ID does not exist" Jul 26 18:24:11.893487 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.893463 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:24:11.895457 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:11.895436 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-614abb29/e2e-lifecycle-model-614abb29-kserve-85dfdc9f95-t2phx"] Jul 26 18:24:12.215719 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:12.215632 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eefc72af-cfce-434c-a442-d632894d1448" path="/var/lib/kubelet/pods/eefc72af-cfce-434c-a442-d632894d1448/volumes" Jul 26 18:24:44.774995 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:44.774952 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:24:44.775491 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:44.775198 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" containerID="cri-o://a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd" gracePeriod=30 Jul 26 18:24:48.763969 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:48.763932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:24:50.038620 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.038554 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:24:50.077235 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.077198 2575 generic.go:356] "Generic (PLEG): container finished" podID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerID="a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd" exitCode=0 Jul 26 18:24:50.077390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.077242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerDied","Data":"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd"} Jul 26 18:24:50.077390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.077284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" event={"ID":"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5","Type":"ContainerDied","Data":"a0b7b16d8dd50b726722af1ed4e45c904ecca710b53376beccd3ce1fdcc6713e"} Jul 26 18:24:50.077390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.077245 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj" Jul 26 18:24:50.077390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.077314 2575 scope.go:117] "RemoveContainer" containerID="a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd" Jul 26 18:24:50.086172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.086153 2575 scope.go:117] "RemoveContainer" containerID="bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d" Jul 26 18:24:50.094205 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.094186 2575 scope.go:117] "RemoveContainer" containerID="a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd" Jul 26 18:24:50.094446 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:24:50.094426 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd\": container with ID starting with a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd not found: ID does not exist" containerID="a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd" Jul 26 18:24:50.094491 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.094457 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd"} err="failed to get container status \"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd\": rpc error: code = NotFound desc = could not find container \"a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd\": container with ID starting with a18822b35db5f01ffc2ea0a6aaacd8a910138effe1b0088486145409a050b2dd not found: ID does not exist" Jul 26 18:24:50.094491 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.094481 2575 scope.go:117] "RemoveContainer" containerID="bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d" Jul 26 18:24:50.094722 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:24:50.094704 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d\": container with ID starting with bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d not found: ID does not exist" containerID="bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d" Jul 26 18:24:50.094775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.094731 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d"} err="failed to get container status \"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d\": rpc error: code = NotFound desc = could not find container \"bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d\": container with ID starting with bb9beb226ac24a1af1e35a173ebd96e496c46b998b3198d2ac874b038b6e070d not found: ID does not exist" Jul 26 18:24:50.105040 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105020 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105123 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105123 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105123 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105104 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105239 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105239 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105255 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbhrd\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105261 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:50.105344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105308 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105509 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105330 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket\") pod \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\" (UID: \"ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5\") " Jul 26 18:24:50.105915 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:50.105915 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:50.105915 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105735 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:24:50.105915 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105868 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.105915 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.105893 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.106234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.106105 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:50.107235 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.107211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:24:50.107456 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.107433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:24:50.107679 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.107653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd" (OuterVolumeSpecName: "kube-api-access-qbhrd") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "kube-api-access-qbhrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:24:50.107679 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.107653 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" (UID: "ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:24:50.206891 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206853 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbhrd\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-kube-api-access-qbhrd\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.206891 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206883 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.206891 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206892 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.207138 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206903 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.207138 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206912 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.207138 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206920 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.207138 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.206928 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:24:50.393401 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.393367 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:24:50.399768 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:50.399741 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-614abb29-openshift-default-575c56fbd4-lpcpj"] Jul 26 18:24:52.208743 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:24:52.208710 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" path="/var/lib/kubelet/pods/ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5/volumes" Jul 26 18:25:00.563397 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:00.563355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:25:40.038821 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.038776 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:25:40.039303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039183 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:40.039303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039200 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:40.039303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039221 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eefc72af-cfce-434c-a442-d632894d1448" containerName="main" Jul 26 18:25:40.039303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039230 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eefc72af-cfce-434c-a442-d632894d1448" containerName="main" Jul 26 18:25:40.039470 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039349 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eefc72af-cfce-434c-a442-d632894d1448" containerName="main" Jul 26 18:25:40.039470 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.039362 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:40.042587 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.042567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.045884 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.045863 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bb8f83-openshift-default-dockercfg-vbrwr\"" Jul 26 18:25:40.053943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.053919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:25:40.105209 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z946w\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105638 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.105638 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.105446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.205926 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.205896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.205948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.205967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.205988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z946w\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206271 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206271 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206271 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206431 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206431 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206601 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206674 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206758 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.206831 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.206800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.207123 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.207094 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.208408 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.208384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.208602 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.208583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.216624 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.216598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.220383 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.220361 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z946w\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w\") pod \"e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.353244 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.353207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:40.494163 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.494132 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:25:40.497057 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:25:40.497031 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f0cb7c_95d0_4141_bc53_915af8f7894c.slice/crio-709c01f8e30a14e623aee025ba733c3d4095bc8fcf221b8232a7a8caca0a9e40 WatchSource:0}: Error finding container 709c01f8e30a14e623aee025ba733c3d4095bc8fcf221b8232a7a8caca0a9e40: Status 404 returned error can't find the container with id 709c01f8e30a14e623aee025ba733c3d4095bc8fcf221b8232a7a8caca0a9e40 Jul 26 18:25:40.499099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.499067 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:40.499152 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.499133 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:40.499191 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:40.499163 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:41.339412 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:41.339356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerStarted","Data":"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a"} Jul 26 18:25:41.339929 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:41.339437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerStarted","Data":"709c01f8e30a14e623aee025ba733c3d4095bc8fcf221b8232a7a8caca0a9e40"} Jul 26 18:25:41.353932 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:41.353908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:41.359988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:41.359835 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:41.360672 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:41.360625 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podStartSLOduration=1.3606038759999999 podStartE2EDuration="1.360603876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:25:41.359053138 +0000 UTC m=+2171.758082175" watchObservedRunningTime="2026-07-26 18:25:41.360603876 +0000 UTC m=+2171.759632922" Jul 26 18:25:42.345538 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:42.345510 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:42.346423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:42.346403 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:25:48.400006 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.399971 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:25:48.400676 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.400643 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:48.400676 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.400667 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:48.400864 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.400739 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc72c98-90b6-4176-ba25-1e0f0ac2c2f5" containerName="istio-proxy" Jul 26 18:25:48.404004 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.403985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.406345 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.406326 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-40c67c-openshift-default-dockercfg-55d2p\"" Jul 26 18:25:48.415854 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.415831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:25:48.490632 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.490584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.490854 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.490707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.490929 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.490903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.490990 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.490952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.491047 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.491028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhfh\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.491132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.491115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.491224 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.491195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.491370 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.491230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.491370 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.491259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592677 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592677 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhfh\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.592993 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.592973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.593361 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.593122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.593361 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.593156 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.593361 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.593326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.593503 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.593384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.593688 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.593669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.595159 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.595134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.595396 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.595374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.601586 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.601563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhfh\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.601670 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.601568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token\") pod \"e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.715311 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.715215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:48.857433 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.857395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:25:48.860819 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:25:48.860782 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bd94f0_7e2d_4577_8ebb_90cf30fc9224.slice/crio-22f67b77049d69cbdcc6b00c24f6f7605caf76f4c8f2246f027276fb70c7c39d WatchSource:0}: Error finding container 22f67b77049d69cbdcc6b00c24f6f7605caf76f4c8f2246f027276fb70c7c39d: Status 404 returned error can't find the container with id 22f67b77049d69cbdcc6b00c24f6f7605caf76f4c8f2246f027276fb70c7c39d Jul 26 18:25:48.863004 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.862971 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:48.863073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.863041 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:48.863073 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:48.863067 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:49.381466 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:49.381424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerStarted","Data":"40a4b60a2499f6eebffaf00faa2a8b7aa9bc75df32b48bc3ffae83bb14183db1"} Jul 26 18:25:49.381675 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:49.381508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerStarted","Data":"22f67b77049d69cbdcc6b00c24f6f7605caf76f4c8f2246f027276fb70c7c39d"} Jul 26 18:25:49.402925 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:49.402863 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podStartSLOduration=1.402847425 podStartE2EDuration="1.402847425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:25:49.39997267 +0000 UTC m=+2179.799001728" watchObservedRunningTime="2026-07-26 18:25:49.402847425 +0000 UTC m=+2179.801876471" Jul 26 18:25:49.715715 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:49.715615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:49.720528 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:49.720501 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:50.386898 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:50.386855 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:50.387948 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:50.387921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:25:50.565820 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:50.565774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:25:58.388033 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:58.387979 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 18:25:58.388562 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:58.388057 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 18:25:59.442621 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.442571 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerID="26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a" exitCode=0 Jul 26 18:25:59.443207 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.442721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerDied","Data":"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a"} Jul 26 18:25:59.443303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.443279 2575 scope.go:117] "RemoveContainer" containerID="26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a" Jul 26 18:25:59.444936 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.444894 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:59.445054 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.444990 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:25:59.445054 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:25:59.445035 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:26:00.454455 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:00.454411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerStarted","Data":"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98"} Jul 26 18:26:01.353914 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:01.353820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:26:01.354616 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:01.354226 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 18:26:01.354616 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:01.354314 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 18:26:01.680738 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:01.680631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:26:02.354083 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:02.354047 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 18:26:02.354362 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:02.354337 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 18:26:03.354114 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:03.354070 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 18:26:03.354692 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:03.354660 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 18:26:04.354397 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:04.354354 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 26 18:26:04.354865 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:04.354430 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 26 18:26:05.357702 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:05.357668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:26:05.358193 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:05.358082 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:26:05.358921 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:05.358898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:26:05.601011 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:05.599457 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:49698->10.134.0.47:15021: read: connection reset by peer" start-of-body= Jul 26 18:26:05.601011 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:05.599549 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:49698->10.134.0.47:15021: read: connection reset by peer" Jul 26 18:26:06.493935 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.493903 2575 generic.go:356] "Generic (PLEG): container finished" podID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerID="40a4b60a2499f6eebffaf00faa2a8b7aa9bc75df32b48bc3ffae83bb14183db1" exitCode=0 Jul 26 18:26:06.494308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.493973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerDied","Data":"40a4b60a2499f6eebffaf00faa2a8b7aa9bc75df32b48bc3ffae83bb14183db1"} Jul 26 18:26:06.494560 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.494547 2575 scope.go:117] "RemoveContainer" containerID="40a4b60a2499f6eebffaf00faa2a8b7aa9bc75df32b48bc3ffae83bb14183db1" Jul 26 18:26:06.495532 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.495504 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:26:06.495648 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.495580 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:26:06.495648 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:06.495620 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:26:07.501509 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:07.501469 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerStarted","Data":"748167238c78d064ca549857a9ccb2f7e09db4d98067eb8ce57201d773e510c1"} Jul 26 18:26:07.715650 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:07.715613 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:26:07.715922 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:07.715893 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 18:26:07.715988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:07.715948 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 18:26:08.716386 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:08.716351 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 18:26:08.716777 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:08.716435 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 18:26:09.715907 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:09.715870 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 18:26:09.716090 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:09.715927 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 18:26:10.715996 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:10.715946 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 26 18:26:10.716404 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:10.716051 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 26 18:26:11.719106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:11.719076 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:26:11.719592 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:11.719376 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:26:11.720148 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:11.720126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:26:16.397502 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.397467 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:26:16.401582 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.401564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.405231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.405201 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb8f83\"/\"test-model-bb8f83-kserve-self-signed-certs\"" Jul 26 18:26:16.405231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.405217 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb8f83\"/\"openshift-service-ca.crt\"" Jul 26 18:26:16.405231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.405229 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb8f83\"/\"kube-root-ca.crt\"" Jul 26 18:26:16.405479 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.405210 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb8f83\"/\"default-dockercfg-gdkv5\"" Jul 26 18:26:16.409316 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.409284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:26:16.488548 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.488726 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.488726 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.488836 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2qr\" (UniqueName: \"kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.488836 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.488836 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.488778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589393 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2qr\" (UniqueName: \"kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589620 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589675 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589730 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.589829 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.589736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.590033 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.590012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.590142 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.590122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.590198 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.590179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.591902 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.591877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.592101 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.592079 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.597119 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.597094 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2qr\" (UniqueName: \"kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr\") pod \"test-model-bb8f83-kserve-8589469b6c-bqgxl\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.712779 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.712686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:16.847642 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:16.847608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:26:16.849040 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:26:16.849009 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac215cbe_21ee_49b0_8a87_eba7dfab01a2.slice/crio-2f4095b322d75739d96bdd3e2e71b086eaed5b102debd2f7d4461824ae2af922 WatchSource:0}: Error finding container 2f4095b322d75739d96bdd3e2e71b086eaed5b102debd2f7d4461824ae2af922: Status 404 returned error can't find the container with id 2f4095b322d75739d96bdd3e2e71b086eaed5b102debd2f7d4461824ae2af922 Jul 26 18:26:17.560613 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:17.560576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" event={"ID":"ac215cbe-21ee-49b0-8a87-eba7dfab01a2","Type":"ContainerStarted","Data":"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f"} Jul 26 18:26:17.560613 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:17.560619 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" event={"ID":"ac215cbe-21ee-49b0-8a87-eba7dfab01a2","Type":"ContainerStarted","Data":"2f4095b322d75739d96bdd3e2e71b086eaed5b102debd2f7d4461824ae2af922"} Jul 26 18:26:17.578715 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:17.578666 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" podStartSLOduration=1.5786519669999999 podStartE2EDuration="1.578651967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:26:17.577095922 +0000 UTC m=+2207.976124967" watchObservedRunningTime="2026-07-26 18:26:17.578651967 +0000 UTC m=+2207.977681012" Jul 26 18:26:20.565987 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:20.565955 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:26:26.713108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:26.713068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:26.713108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:26.713120 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:26.726206 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:26.726176 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:27.621308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:27.621279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:26:48.379081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:26:48.379042 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:27:01.708424 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:01.708324 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:27:25.505290 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:25.505243 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:27:31.263085 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:31.263048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:27:40.525740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.525673 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" podUID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" containerName="main" containerID="cri-o://1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f" gracePeriod=45 Jul 26 18:27:40.786778 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.786713 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:27:40.941730 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.941696 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.941942 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.941761 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.941942 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.941838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.942084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.941962 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.942084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home" (OuterVolumeSpecName: "home") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:40.942084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942031 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.942226 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942079 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:40.942226 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942121 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v2qr\" (UniqueName: \"kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr\") pod \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\" (UID: \"ac215cbe-21ee-49b0-8a87-eba7dfab01a2\") " Jul 26 18:27:40.942300 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942244 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:40.942512 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942491 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:40.942512 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942514 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:40.942665 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.942529 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:40.944082 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.944060 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:27:40.944182 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.944109 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm" (OuterVolumeSpecName: "dshm") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:40.944182 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.944161 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr" (OuterVolumeSpecName: "kube-api-access-5v2qr") pod "ac215cbe-21ee-49b0-8a87-eba7dfab01a2" (UID: "ac215cbe-21ee-49b0-8a87-eba7dfab01a2"). InnerVolumeSpecName "kube-api-access-5v2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:27:40.993591 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.993555 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" containerID="1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f" exitCode=0 Jul 26 18:27:40.993773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.993603 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" Jul 26 18:27:40.993773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.993609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" event={"ID":"ac215cbe-21ee-49b0-8a87-eba7dfab01a2","Type":"ContainerDied","Data":"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f"} Jul 26 18:27:40.993773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.993724 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl" event={"ID":"ac215cbe-21ee-49b0-8a87-eba7dfab01a2","Type":"ContainerDied","Data":"2f4095b322d75739d96bdd3e2e71b086eaed5b102debd2f7d4461824ae2af922"} Jul 26 18:27:40.993773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:40.993757 2575 scope.go:117] "RemoveContainer" containerID="1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f" Jul 26 18:27:41.003132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.003107 2575 scope.go:117] "RemoveContainer" containerID="1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f" Jul 26 18:27:41.003408 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:27:41.003387 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f\": container with ID starting with 1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f not found: ID does not exist" containerID="1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f" Jul 26 18:27:41.003461 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.003418 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f"} err="failed to get container status \"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f\": rpc error: code = NotFound desc = could not find container \"1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f\": container with ID starting with 1425664baf7791b8d1e6836a013b611a1275d493e903180260f574bf6e12965f not found: ID does not exist" Jul 26 18:27:41.015775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.015745 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:27:41.022472 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.022447 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bb8f83/test-model-bb8f83-kserve-8589469b6c-bqgxl"] Jul 26 18:27:41.044018 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.043954 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:41.044018 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.043977 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2qr\" (UniqueName: \"kubernetes.io/projected/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-kube-api-access-5v2qr\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:41.044018 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:41.043987 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac215cbe-21ee-49b0-8a87-eba7dfab01a2-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:42.208655 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:42.208621 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" path="/var/lib/kubelet/pods/ac215cbe-21ee-49b0-8a87-eba7dfab01a2/volumes" Jul 26 18:27:46.164274 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:46.164232 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:27:48.104775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:48.104737 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:27:48.105262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:48.105011 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" containerID="cri-o://be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98" gracePeriod=30 Jul 26 18:27:50.360149 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:50.360105 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 18:27:50.360513 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:50.360176 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 18:27:53.364729 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.364704 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:27:53.459122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459058 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z946w\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459079 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459207 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459233 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459260 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459515 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459343 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token\") pod \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\" (UID: \"f3f0cb7c-95d0-4141-bc53-915af8f7894c\") " Jul 26 18:27:53.459717 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459550 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:53.459717 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459686 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:53.459854 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459735 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.459854 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459809 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:53.459955 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459911 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:27:53.460027 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.459998 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:53.461443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.461411 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w" (OuterVolumeSpecName: "kube-api-access-z946w") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "kube-api-access-z946w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:27:53.461443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.461409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:27:53.461597 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.461441 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:27:53.461773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.461754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3f0cb7c-95d0-4141-bc53-915af8f7894c" (UID: "f3f0cb7c-95d0-4141-bc53-915af8f7894c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:27:53.561300 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561261 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561300 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561292 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z946w\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-kube-api-access-z946w\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561300 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561303 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561550 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561313 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561550 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561321 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561550 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561330 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f0cb7c-95d0-4141-bc53-915af8f7894c-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561550 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561338 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:53.561550 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:53.561346 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f0cb7c-95d0-4141-bc53-915af8f7894c-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:27:54.061671 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.061637 2575 generic.go:356] "Generic (PLEG): container finished" podID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerID="be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98" exitCode=0 Jul 26 18:27:54.061869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.061708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerDied","Data":"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98"} Jul 26 18:27:54.061869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.061781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" event={"ID":"f3f0cb7c-95d0-4141-bc53-915af8f7894c","Type":"ContainerDied","Data":"709c01f8e30a14e623aee025ba733c3d4095bc8fcf221b8232a7a8caca0a9e40"} Jul 26 18:27:54.061869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.061708 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh" Jul 26 18:27:54.061869 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.061824 2575 scope.go:117] "RemoveContainer" containerID="be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98" Jul 26 18:27:54.071626 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.071609 2575 scope.go:117] "RemoveContainer" containerID="26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a" Jul 26 18:27:54.079717 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.079697 2575 scope.go:117] "RemoveContainer" containerID="be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98" Jul 26 18:27:54.080021 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:27:54.080001 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98\": container with ID starting with be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98 not found: ID does not exist" containerID="be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98" Jul 26 18:27:54.080094 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.080034 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98"} err="failed to get container status \"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98\": rpc error: code = NotFound desc = could not find container \"be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98\": container with ID starting with be73c59cad7e9469a6d555d8329d2badf0f2413cc4ec53fd06055be6e7aa4a98 not found: ID does not exist" Jul 26 18:27:54.080094 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.080060 2575 scope.go:117] "RemoveContainer" containerID="26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a" Jul 26 18:27:54.080306 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:27:54.080288 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a\": container with ID starting with 26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a not found: ID does not exist" containerID="26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a" Jul 26 18:27:54.080347 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.080314 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a"} err="failed to get container status \"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a\": rpc error: code = NotFound desc = could not find container \"26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a\": container with ID starting with 26d959e16085bb5c38413ee2899ae83687d76e460a314441e698c07c09f00c4a not found: ID does not exist" Jul 26 18:27:54.083902 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.083878 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:27:54.088077 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.088059 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bb8f83-openshift-default-86f64c7d5b-pdmrh"] Jul 26 18:27:54.208561 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:27:54.208530 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" path="/var/lib/kubelet/pods/f3f0cb7c-95d0-4141-bc53-915af8f7894c/volumes" Jul 26 18:28:01.758201 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:01.758155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:28:10.053669 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.053632 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:28:10.054172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.053913 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" containerID="cri-o://748167238c78d064ca549857a9ccb2f7e09db4d98067eb8ce57201d773e510c1" gracePeriod=30 Jul 26 18:28:10.890693 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.890659 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:28:10.891127 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891111 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" containerName="main" Jul 26 18:28:10.891197 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891129 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" containerName="main" Jul 26 18:28:10.891197 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891148 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:10.891197 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891154 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:10.891301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891233 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:10.891301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891244 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:10.891301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.891253 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac215cbe-21ee-49b0-8a87-eba7dfab01a2" containerName="main" Jul 26 18:28:10.894782 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.894761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:10.897196 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.897176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b6e870-openshift-default-dockercfg-6hn8g\"" Jul 26 18:28:10.905436 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:10.905413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:28:11.044619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.044815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.044815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.044815 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.044947 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46trr\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.044947 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.045020 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.045020 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.045020 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.044990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.145682 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.145682 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46trr\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.145933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146208 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146760 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146760 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.146760 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.146467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.148526 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.148500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.148630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.148584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.154111 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.154080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.154243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.154223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46trr\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr\") pod \"e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.205854 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.205813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:11.347811 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.347761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:28:11.349114 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:28:11.349087 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e176b7_3895_4ca7_8b16_07faea139af6.slice/crio-bfbf507d2dfb687180e9f456e375df26b71340cfd1e7400f37a6c70125ba9ad3 WatchSource:0}: Error finding container bfbf507d2dfb687180e9f456e375df26b71340cfd1e7400f37a6c70125ba9ad3: Status 404 returned error can't find the container with id bfbf507d2dfb687180e9f456e375df26b71340cfd1e7400f37a6c70125ba9ad3 Jul 26 18:28:11.351492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.351451 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:11.351580 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.351544 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:11.351621 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.351588 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:11.722576 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.722490 2575 patch_prober.go:28] interesting pod/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 18:28:11.722576 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:11.722552 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 18:28:12.159928 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:12.159894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerStarted","Data":"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0"} Jul 26 18:28:12.160276 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:12.159985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerStarted","Data":"bfbf507d2dfb687180e9f456e375df26b71340cfd1e7400f37a6c70125ba9ad3"} Jul 26 18:28:12.180526 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:12.180473 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podStartSLOduration=2.180457888 podStartE2EDuration="2.180457888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:28:12.179590699 +0000 UTC m=+2322.578619754" watchObservedRunningTime="2026-07-26 18:28:12.180457888 +0000 UTC m=+2322.579486933" Jul 26 18:28:12.209488 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:12.209456 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:12.211078 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:12.211055 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:13.165681 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:13.165646 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:13.166680 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:13.166661 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:15.178977 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.178944 2575 generic.go:356] "Generic (PLEG): container finished" podID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerID="748167238c78d064ca549857a9ccb2f7e09db4d98067eb8ce57201d773e510c1" exitCode=0 Jul 26 18:28:15.179367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.179013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerDied","Data":"748167238c78d064ca549857a9ccb2f7e09db4d98067eb8ce57201d773e510c1"} Jul 26 18:28:15.179367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.179078 2575 scope.go:117] "RemoveContainer" containerID="40a4b60a2499f6eebffaf00faa2a8b7aa9bc75df32b48bc3ffae83bb14183db1" Jul 26 18:28:15.319326 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.319264 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:28:15.391446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391410 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391529 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391578 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391697 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fhfh\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391803 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.391923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391696 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:28:15.391923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391831 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket\") pod \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\" (UID: \"02bd94f0-7e2d-4577-8ebb-90cf30fc9224\") " Jul 26 18:28:15.392180 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391931 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:28:15.392180 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.391979 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:28:15.392180 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392126 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:28:15.392330 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392299 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.392330 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392321 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.392422 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392336 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.392422 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392350 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.392715 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.392689 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data" (OuterVolumeSpecName: "istio-data") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:28:15.393904 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.393880 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token" (OuterVolumeSpecName: "istio-token") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:28:15.394003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.393904 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:28:15.394144 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.394127 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:28:15.394203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.394171 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh" (OuterVolumeSpecName: "kube-api-access-8fhfh") pod "02bd94f0-7e2d-4577-8ebb-90cf30fc9224" (UID: "02bd94f0-7e2d-4577-8ebb-90cf30fc9224"). InnerVolumeSpecName "kube-api-access-8fhfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:28:15.493561 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.493528 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.493561 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.493559 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.493561 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.493569 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.493884 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.493578 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhfh\" (UniqueName: \"kubernetes.io/projected/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-kube-api-access-8fhfh\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:15.493884 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:15.493587 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02bd94f0-7e2d-4577-8ebb-90cf30fc9224-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:28:16.186480 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:16.186441 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" Jul 26 18:28:16.186955 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:16.186447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds" event={"ID":"02bd94f0-7e2d-4577-8ebb-90cf30fc9224","Type":"ContainerDied","Data":"22f67b77049d69cbdcc6b00c24f6f7605caf76f4c8f2246f027276fb70c7c39d"} Jul 26 18:28:16.186955 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:16.186585 2575 scope.go:117] "RemoveContainer" containerID="748167238c78d064ca549857a9ccb2f7e09db4d98067eb8ce57201d773e510c1" Jul 26 18:28:16.209351 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:16.209297 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:28:16.211694 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:16.211670 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-40c67c-openshift-default-77d87d98c8-96kds"] Jul 26 18:28:18.209254 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:18.209218 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" path="/var/lib/kubelet/pods/02bd94f0-7e2d-4577-8ebb-90cf30fc9224/volumes" Jul 26 18:28:19.278898 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.278863 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:28:19.279506 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279489 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:28:19.279564 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:28:19.279564 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279529 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:19.279564 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279538 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f0cb7c-95d0-4141-bc53-915af8f7894c" containerName="istio-proxy" Jul 26 18:28:19.279663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279627 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:28:19.279663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.279642 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:28:19.284673 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.284647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.287085 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.287064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-167468-openshift-default-dockercfg-jglpw\"" Jul 26 18:28:19.293506 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.293480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:28:19.338585 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.338818 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.338818 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.338818 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.338818 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.339049 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgxt\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.339049 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.339049 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.339049 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.338987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.439634 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.439842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.439842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.439842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgxt\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.439842 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439817 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.439893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440170 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440266 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440349 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440418 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440481 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440481 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440591 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.440700 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.440672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.442633 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.442615 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.442953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.442931 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.448021 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.447996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.448145 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.448125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgxt\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt\") pod \"e2e-api-b-167468-openshift-default-66c857cf94-qhs9c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.596113 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.596014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:19.736984 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.736958 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:28:19.739859 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:28:19.739833 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad992b7f_7111_4a34_b855_dcc4ecce017c.slice/crio-4ebf0795474a4ca3f6b869e99b5464635cf7c69abbc9e4f831a6c3d57ce6d023 WatchSource:0}: Error finding container 4ebf0795474a4ca3f6b869e99b5464635cf7c69abbc9e4f831a6c3d57ce6d023: Status 404 returned error can't find the container with id 4ebf0795474a4ca3f6b869e99b5464635cf7c69abbc9e4f831a6c3d57ce6d023 Jul 26 18:28:19.742551 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.742518 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:19.742655 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.742596 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:19.742655 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:19.742641 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:20.211707 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:20.211672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerStarted","Data":"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6"} Jul 26 18:28:20.211893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:20.211725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerStarted","Data":"4ebf0795474a4ca3f6b869e99b5464635cf7c69abbc9e4f831a6c3d57ce6d023"} Jul 26 18:28:20.232742 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:20.232682 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podStartSLOduration=1.232664665 podStartE2EDuration="1.232664665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:28:20.232093024 +0000 UTC m=+2330.631122069" watchObservedRunningTime="2026-07-26 18:28:20.232664665 +0000 UTC m=+2330.631693710" Jul 26 18:28:20.596552 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:20.596464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:20.601394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:20.601368 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:21.217129 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:21.217097 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:21.218232 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:21.218209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:21.766052 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:21.766012 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:28:29.206960 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:29.206922 2575 patch_prober.go:28] interesting pod/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 18:28:29.207438 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:29.206983 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 18:28:30.268851 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.268812 2575 generic.go:356] "Generic (PLEG): container finished" podID="65e176b7-3895-4ca7-8b16-07faea139af6" containerID="867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0" exitCode=0 Jul 26 18:28:30.269235 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.268898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerDied","Data":"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0"} Jul 26 18:28:30.269293 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.269269 2575 scope.go:117] "RemoveContainer" containerID="867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0" Jul 26 18:28:30.270309 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.270278 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:30.270368 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.270350 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:30.270408 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:30.270379 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:31.280537 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:31.280494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerStarted","Data":"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600"} Jul 26 18:28:32.206221 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:32.206186 2575 patch_prober.go:28] interesting pod/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 26 18:28:32.206448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:32.206245 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 26 18:28:32.209457 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:32.209437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:33.206976 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:33.206940 2575 patch_prober.go:28] interesting pod/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 26 18:28:33.207509 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:33.207480 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 26 18:28:34.206390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:34.206351 2575 patch_prober.go:28] interesting pod/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 26 18:28:34.206579 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:34.206400 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 26 18:28:35.206412 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:35.206372 2575 patch_prober.go:28] interesting pod/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 26 18:28:35.206827 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:35.206427 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 26 18:28:36.210615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:36.210585 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:36.211071 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:36.210846 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:36.211615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:36.211593 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:28:37.259248 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:37.258949 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 18:28:37.259248 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:37.259014 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 18:28:37.476144 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:37.476102 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:28:39.327710 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.327676 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerID="b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6" exitCode=0 Jul 26 18:28:39.328161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.327761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerDied","Data":"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6"} Jul 26 18:28:39.328219 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.328205 2575 scope.go:117] "RemoveContainer" containerID="b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6" Jul 26 18:28:39.329303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.329273 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:39.329406 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.329341 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:39.329406 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:39.329373 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:28:40.334298 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:40.334258 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerStarted","Data":"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd"} Jul 26 18:28:40.596529 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:40.596446 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:40.596679 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:40.596641 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 18:28:40.596721 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:40.596687 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 18:28:41.596384 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:41.596347 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 18:28:41.596774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:41.596415 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 18:28:42.597013 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:42.596972 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 18:28:42.597501 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:42.597042 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 18:28:43.596387 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:43.596350 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 26 18:28:43.596570 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:43.596407 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 26 18:28:44.600262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:44.600232 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:44.600663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:44.600497 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:44.601171 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:44.601157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:28:53.462886 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:28:53.462846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:29:00.892663 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:00.892626 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:29:00.893134 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:00.892917 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" containerID="cri-o://d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600" gracePeriod=30 Jul 26 18:29:06.145422 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.145394 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:29:06.230957 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.230869 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.230957 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.230945 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.230979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46trr\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231172 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231169 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231195 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231203 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:06.231388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231245 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231283 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data\") pod \"65e176b7-3895-4ca7-8b16-07faea139af6\" (UID: \"65e176b7-3895-4ca7-8b16-07faea139af6\") " Jul 26 18:29:06.231619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:06.231619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231423 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:06.231733 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231715 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.231810 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231741 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.231810 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231757 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.231920 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.231776 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:29:06.232363 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.232335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data" (OuterVolumeSpecName: "istio-data") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:06.233328 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.233299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr" (OuterVolumeSpecName: "kube-api-access-46trr") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "kube-api-access-46trr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:29:06.233328 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.233300 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:29:06.233460 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.233297 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token" (OuterVolumeSpecName: "istio-token") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:29:06.233460 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.233297 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65e176b7-3895-4ca7-8b16-07faea139af6" (UID: "65e176b7-3895-4ca7-8b16-07faea139af6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:06.332490 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332448 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.332490 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332485 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e176b7-3895-4ca7-8b16-07faea139af6-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.332490 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332497 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.332738 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332507 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e176b7-3895-4ca7-8b16-07faea139af6-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.332738 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332520 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e176b7-3895-4ca7-8b16-07faea139af6-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.332738 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.332532 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46trr\" (UniqueName: \"kubernetes.io/projected/65e176b7-3895-4ca7-8b16-07faea139af6-kube-api-access-46trr\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:06.494003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.493923 2575 generic.go:356] "Generic (PLEG): container finished" podID="65e176b7-3895-4ca7-8b16-07faea139af6" containerID="d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600" exitCode=0 Jul 26 18:29:06.494003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.493973 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" Jul 26 18:29:06.494183 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.494002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerDied","Data":"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600"} Jul 26 18:29:06.494183 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.494062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4" event={"ID":"65e176b7-3895-4ca7-8b16-07faea139af6","Type":"ContainerDied","Data":"bfbf507d2dfb687180e9f456e375df26b71340cfd1e7400f37a6c70125ba9ad3"} Jul 26 18:29:06.494183 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.494092 2575 scope.go:117] "RemoveContainer" containerID="d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600" Jul 26 18:29:06.503425 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.503406 2575 scope.go:117] "RemoveContainer" containerID="867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0" Jul 26 18:29:06.511626 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.511604 2575 scope.go:117] "RemoveContainer" containerID="d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600" Jul 26 18:29:06.511926 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:29:06.511904 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600\": container with ID starting with d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600 not found: ID does not exist" containerID="d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600" Jul 26 18:29:06.511991 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.511934 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600"} err="failed to get container status \"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600\": rpc error: code = NotFound desc = could not find container \"d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600\": container with ID starting with d063fe32e36851c9ce2d8a3555cb70d759d28cfe18392de3d9e701f5dd2a1600 not found: ID does not exist" Jul 26 18:29:06.511991 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.511951 2575 scope.go:117] "RemoveContainer" containerID="867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0" Jul 26 18:29:06.512193 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:29:06.512174 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0\": container with ID starting with 867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0 not found: ID does not exist" containerID="867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0" Jul 26 18:29:06.512232 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.512200 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0"} err="failed to get container status \"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0\": rpc error: code = NotFound desc = could not find container \"867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0\": container with ID starting with 867aa644524d1b6c369b7bab20918702c56497810134dd0689b5a6bf467d3cc0 not found: ID does not exist" Jul 26 18:29:06.516804 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.516767 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:29:06.520138 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:06.520117 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b6e870-openshift-default-68899dfcbd-4rzv4"] Jul 26 18:29:08.209142 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:08.209111 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" path="/var/lib/kubelet/pods/65e176b7-3895-4ca7-8b16-07faea139af6/volumes" Jul 26 18:29:21.771189 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:21.771149 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:29:27.917615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:27.917582 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:29:27.918147 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:27.917808 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" containerID="cri-o://7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd" gracePeriod=30 Jul 26 18:29:28.753975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.753941 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:29:28.754620 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754585 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.754620 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754610 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.754775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.754775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754634 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.754775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754664 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:29:28.754775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754671 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="02bd94f0-7e2d-4577-8ebb-90cf30fc9224" containerName="istio-proxy" Jul 26 18:29:28.754775 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754766 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.754990 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.754778 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65e176b7-3895-4ca7-8b16-07faea139af6" containerName="istio-proxy" Jul 26 18:29:28.760035 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.760015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.762730 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.762705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-57f93c-openshift-default-dockercfg-hvxb4\"" Jul 26 18:29:28.770207 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.770179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:29:28.849965 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.849925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.849965 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.849970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.849995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ct9g\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850158 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.850234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.850190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951411 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct9g\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.951924 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.952306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.951984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.952306 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.952028 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.952504 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.952485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.953696 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.953674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.954199 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.954183 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.959845 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.959818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:28.960051 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:28.960036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct9g\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g\") pod \"e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:29.071826 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.071710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:29.216782 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.216752 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:29:29.218556 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:29:29.218527 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0e5bf6_a3a8_4044_8597_9559ed2dfc09.slice/crio-ae4cbf7911ab8b8b093bcabdcf1d00b9cdc174c6145562badbe9060a4761be30 WatchSource:0}: Error finding container ae4cbf7911ab8b8b093bcabdcf1d00b9cdc174c6145562badbe9060a4761be30: Status 404 returned error can't find the container with id ae4cbf7911ab8b8b093bcabdcf1d00b9cdc174c6145562badbe9060a4761be30 Jul 26 18:29:29.221111 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.221072 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:29.221203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.221144 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:29.221203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.221176 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:29.603970 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.603909 2575 patch_prober.go:28] interesting pod/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 18:29:29.604178 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.604001 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 18:29:29.619071 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.619031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerStarted","Data":"764c68502cbfd1774f81b0ddb553acccf4becd343ca64e8a9b8586567b2c285d"} Jul 26 18:29:29.619237 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.619084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerStarted","Data":"ae4cbf7911ab8b8b093bcabdcf1d00b9cdc174c6145562badbe9060a4761be30"} Jul 26 18:29:29.639684 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:29.639626 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podStartSLOduration=1.63960645 podStartE2EDuration="1.63960645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:29:29.6373932 +0000 UTC m=+2400.036422282" watchObservedRunningTime="2026-07-26 18:29:29.63960645 +0000 UTC m=+2400.038635570" Jul 26 18:29:30.072649 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:30.072565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:30.077493 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:30.077467 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:30.624390 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:30.624303 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:30.625286 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:30.625267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:33.165777 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.165746 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:29:33.300132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300047 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300090 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300210 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300236 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300266 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300293 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgxt\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300349 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300388 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300381 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket\") pod \"ad992b7f-7111-4a34-b855-dcc4ecce017c\" (UID: \"ad992b7f-7111-4a34-b855-dcc4ecce017c\") " Jul 26 18:29:33.300697 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300545 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:33.300697 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300587 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:29:33.300843 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300693 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:33.300843 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300770 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:33.300953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300904 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad992b7f-7111-4a34-b855-dcc4ecce017c-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.300953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300922 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.300953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300936 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.300953 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.300947 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.301273 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.301249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:33.302770 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.302741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:29:33.302770 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.302751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:29:33.302935 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.302743 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt" (OuterVolumeSpecName: "kube-api-access-5jgxt") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "kube-api-access-5jgxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:29:33.302935 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.302782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad992b7f-7111-4a34-b855-dcc4ecce017c" (UID: "ad992b7f-7111-4a34-b855-dcc4ecce017c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:29:33.402511 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.402474 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.402511 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.402504 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.402511 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.402514 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.402511 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.402522 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad992b7f-7111-4a34-b855-dcc4ecce017c-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.402783 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.402531 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgxt\" (UniqueName: \"kubernetes.io/projected/ad992b7f-7111-4a34-b855-dcc4ecce017c-kube-api-access-5jgxt\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:29:33.641345 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.641309 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerID="7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd" exitCode=0 Jul 26 18:29:33.641549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.641356 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" Jul 26 18:29:33.641549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.641396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerDied","Data":"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd"} Jul 26 18:29:33.641549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.641465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c" event={"ID":"ad992b7f-7111-4a34-b855-dcc4ecce017c","Type":"ContainerDied","Data":"4ebf0795474a4ca3f6b869e99b5464635cf7c69abbc9e4f831a6c3d57ce6d023"} Jul 26 18:29:33.641549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.641496 2575 scope.go:117] "RemoveContainer" containerID="7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd" Jul 26 18:29:33.651403 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.651381 2575 scope.go:117] "RemoveContainer" containerID="b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6" Jul 26 18:29:33.660152 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.660133 2575 scope.go:117] "RemoveContainer" containerID="7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd" Jul 26 18:29:33.660416 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:29:33.660395 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd\": container with ID starting with 7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd not found: ID does not exist" containerID="7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd" Jul 26 18:29:33.660487 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.660425 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd"} err="failed to get container status \"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd\": rpc error: code = NotFound desc = could not find container \"7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd\": container with ID starting with 7c82cc8fa9c5464864d41fb6db943315d32aae2376da2e8db75f1ed7d535e6cd not found: ID does not exist" Jul 26 18:29:33.660487 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.660444 2575 scope.go:117] "RemoveContainer" containerID="b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6" Jul 26 18:29:33.660704 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:29:33.660686 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6\": container with ID starting with b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6 not found: ID does not exist" containerID="b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6" Jul 26 18:29:33.660751 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.660709 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6"} err="failed to get container status \"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6\": rpc error: code = NotFound desc = could not find container \"b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6\": container with ID starting with b8dd4ff2e4e2a7aaf5413911b0aa17c949377d8f9fab4fdfd1c348679c5e0cf6 not found: ID does not exist" Jul 26 18:29:33.668564 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.668511 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:29:33.670615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:33.670596 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-167468-openshift-default-66c857cf94-qhs9c"] Jul 26 18:29:34.210110 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:34.210076 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" path="/var/lib/kubelet/pods/ad992b7f-7111-4a34-b855-dcc4ecce017c/volumes" Jul 26 18:29:36.631960 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.631929 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:29:36.632474 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632451 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.632641 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632484 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.632641 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632524 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.632641 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632535 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.632829 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632665 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.632829 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.632682 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad992b7f-7111-4a34-b855-dcc4ecce017c" containerName="istio-proxy" Jul 26 18:29:36.635923 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.635901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.638389 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.638368 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6aa894-openshift-default-dockercfg-6p6xk\"" Jul 26 18:29:36.647416 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.647390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:29:36.736233 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736233 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736238 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.736514 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.736508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kx46\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837434 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837434 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kx46\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837661 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837661 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837661 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837814 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837860 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837908 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.837908 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.838005 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.837938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.838220 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.838189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.838312 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.838242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.838312 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.838295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.838412 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.838371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.840292 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.840269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.840398 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.840279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.845376 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.845350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.845376 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.845365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kx46\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46\") pod \"e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.947072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.946977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:36.975122 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:36.975091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:29:37.095260 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.095230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:29:37.097265 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:29:37.097235 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56531475_5229_48c9_994a_aa939302213a.slice/crio-158e34a2e4ae26cb08fceb5ac3d50164ad04f7594e6ae850176e51d51541d927 WatchSource:0}: Error finding container 158e34a2e4ae26cb08fceb5ac3d50164ad04f7594e6ae850176e51d51541d927: Status 404 returned error can't find the container with id 158e34a2e4ae26cb08fceb5ac3d50164ad04f7594e6ae850176e51d51541d927 Jul 26 18:29:37.099799 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.099756 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:37.099873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.099837 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:37.099910 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.099870 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:37.663988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.663947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerStarted","Data":"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784"} Jul 26 18:29:37.664480 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.664013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerStarted","Data":"158e34a2e4ae26cb08fceb5ac3d50164ad04f7594e6ae850176e51d51541d927"} Jul 26 18:29:37.684685 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.684624 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podStartSLOduration=1.684609305 podStartE2EDuration="1.684609305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:29:37.683024444 +0000 UTC m=+2408.082053491" watchObservedRunningTime="2026-07-26 18:29:37.684609305 +0000 UTC m=+2408.083638350" Jul 26 18:29:37.948200 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:37.948110 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:38.958202 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:38.958165 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:39.675368 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:39.675332 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:39.676419 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:39.676400 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:45.625558 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.625519 2575 patch_prober.go:28] interesting pod/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 26 18:29:45.626099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.625597 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 26 18:29:45.711318 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.711284 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerID="764c68502cbfd1774f81b0ddb553acccf4becd343ca64e8a9b8586567b2c285d" exitCode=0 Jul 26 18:29:45.711490 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.711362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerDied","Data":"764c68502cbfd1774f81b0ddb553acccf4becd343ca64e8a9b8586567b2c285d"} Jul 26 18:29:45.711785 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.711771 2575 scope.go:117] "RemoveContainer" containerID="764c68502cbfd1774f81b0ddb553acccf4becd343ca64e8a9b8586567b2c285d" Jul 26 18:29:45.712921 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.712895 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:45.712987 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.712962 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:45.713031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:45.712993 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:46.058014 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:46.057919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:29:46.718473 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:46.718437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerStarted","Data":"1a5a1d3be7d726cd0b5f97ff2e5b4e8a9c519ccae4a45845aedd455121d7f6e2"} Jul 26 18:29:47.073165 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:47.073059 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:47.073360 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:47.073196 2575 patch_prober.go:28] interesting pod/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 26 18:29:47.073360 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:47.073245 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 26 18:29:48.072670 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:48.072620 2575 patch_prober.go:28] interesting pod/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 26 18:29:48.073222 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:48.072697 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 26 18:29:49.073108 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:49.073059 2575 patch_prober.go:28] interesting pod/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 26 18:29:49.073615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:49.073136 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 26 18:29:50.072302 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:50.072255 2575 patch_prober.go:28] interesting pod/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 26 18:29:50.072527 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:50.072324 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 26 18:29:51.076919 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:51.076885 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:51.077364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:51.077165 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:51.077949 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:51.077933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:29:54.677013 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.676971 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 18:29:54.677504 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.677049 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 18:29:54.764596 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.764562 2575 generic.go:356] "Generic (PLEG): container finished" podID="56531475-5229-48c9-994a-aa939302213a" containerID="f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784" exitCode=0 Jul 26 18:29:54.764759 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.764613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerDied","Data":"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784"} Jul 26 18:29:54.765128 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.765112 2575 scope.go:117] "RemoveContainer" containerID="f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784" Jul 26 18:29:54.766253 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.766223 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:54.766310 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.766298 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:54.766351 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:54.766332 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:29:55.771555 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:55.771515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerStarted","Data":"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b"} Jul 26 18:29:55.947546 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:55.947500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:55.947862 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:55.947680 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 18:29:55.948032 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:55.947999 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 18:29:56.948141 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:56.948100 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 18:29:56.948727 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:56.948694 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 18:29:57.947781 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:57.947739 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 18:29:57.948102 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:57.948068 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 18:29:58.948009 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:58.947970 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 26 18:29:58.948685 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:58.948650 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 26 18:29:59.951943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:59.951912 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:59.952415 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:59.952300 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:29:59.953022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:29:59.953002 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:30:07.174008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:07.173967 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:30:53.902162 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.902122 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:30:53.906658 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.906635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:53.909364 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.909343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6aa894\"/\"openshift-service-ca.crt\"" Jul 26 18:30:53.910336 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.910311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6aa894\"/\"kube-root-ca.crt\"" Jul 26 18:30:53.910571 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.910546 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6aa894\"/\"default-dockercfg-qln6h\"" Jul 26 18:30:53.910674 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.910591 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6aa894\"/\"auth-test-model-6aa894-kserve-self-signed-certs\"" Jul 26 18:30:53.916946 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:53.916912 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:30:54.024688 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xn6\" (UniqueName: \"kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.024906 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.024906 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.024906 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.025038 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.025038 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.024944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125669 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125773 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xn6\" (UniqueName: \"kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.125893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.125872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.126203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.126093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.126203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.126101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.126203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.126172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.128022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.127996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.128394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.128372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.135184 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.135158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xn6\" (UniqueName: \"kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6\") pod \"auth-test-model-6aa894-kserve-5fd5778479-wln9n\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.216200 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.216116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:30:54.357344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:54.357319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:30:54.359985 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:30:54.359959 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2310a8_9593_47e1_8b68_1a300ea6acd2.slice/crio-e3af71223fbdfa4d9495e2d4aeed003825312cfcd0be6d76bffc27775b2cf9ec WatchSource:0}: Error finding container e3af71223fbdfa4d9495e2d4aeed003825312cfcd0be6d76bffc27775b2cf9ec: Status 404 returned error can't find the container with id e3af71223fbdfa4d9495e2d4aeed003825312cfcd0be6d76bffc27775b2cf9ec Jul 26 18:30:55.079844 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:55.079805 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" event={"ID":"0c2310a8-9593-47e1-8b68-1a300ea6acd2","Type":"ContainerStarted","Data":"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b"} Jul 26 18:30:55.080253 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:55.079854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" event={"ID":"0c2310a8-9593-47e1-8b68-1a300ea6acd2","Type":"ContainerStarted","Data":"e3af71223fbdfa4d9495e2d4aeed003825312cfcd0be6d76bffc27775b2cf9ec"} Jul 26 18:30:55.100597 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:55.100541 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" podStartSLOduration=2.100523856 podStartE2EDuration="2.100523856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:30:55.098571156 +0000 UTC m=+2485.497600225" watchObservedRunningTime="2026-07-26 18:30:55.100523856 +0000 UTC m=+2485.499552900" Jul 26 18:30:57.468981 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:30:57.468941 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:31:04.216759 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:04.216725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:31:04.217250 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:04.216778 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:31:04.228943 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:04.228919 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:31:05.139376 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:05.139337 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:31:21.967013 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:21.966918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:31:41.863957 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:31:41.863919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:32:00.077650 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:32:00.077615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:32:26.767614 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:32:26.767575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:32:46.498235 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:32:46.498196 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:33:08.277946 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:33:08.277859 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:33:26.808584 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:33:26.808548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:33:48.377458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:33:48.377413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:34:04.665926 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:34:04.665890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:34:26.263287 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:34:26.263202 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:34:41.773246 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:34:41.773207 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:35:02.286307 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:02.286266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:35:05.904007 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:05.903972 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:35:17.867506 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:17.867454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:35:20.923615 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:20.923573 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" podUID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" containerName="main" containerID="cri-o://03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b" gracePeriod=45 Jul 26 18:35:21.176520 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.176458 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:35:21.297938 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.297900 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.297968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298033 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298135 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298121 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298345 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298165 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7xn6\" (UniqueName: \"kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6\") pod \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\" (UID: \"0c2310a8-9593-47e1-8b68-1a300ea6acd2\") " Jul 26 18:35:21.298345 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298222 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:35:21.298345 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298265 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:35:21.298467 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298354 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home" (OuterVolumeSpecName: "home") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:35:21.298589 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298562 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.298648 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298598 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.298648 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.298613 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.300366 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.300338 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm" (OuterVolumeSpecName: "dshm") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:35:21.300366 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.300352 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6" (OuterVolumeSpecName: "kube-api-access-s7xn6") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "kube-api-access-s7xn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:35:21.300524 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.300359 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c2310a8-9593-47e1-8b68-1a300ea6acd2" (UID: "0c2310a8-9593-47e1-8b68-1a300ea6acd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:35:21.399873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.399832 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2310a8-9593-47e1-8b68-1a300ea6acd2-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.399873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.399866 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7xn6\" (UniqueName: \"kubernetes.io/projected/0c2310a8-9593-47e1-8b68-1a300ea6acd2-kube-api-access-s7xn6\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.399873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.399876 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2310a8-9593-47e1-8b68-1a300ea6acd2-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:35:21.484458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.484372 2575 generic.go:356] "Generic (PLEG): container finished" podID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" containerID="03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b" exitCode=0 Jul 26 18:35:21.484458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.484413 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" Jul 26 18:35:21.484629 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.484468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" event={"ID":"0c2310a8-9593-47e1-8b68-1a300ea6acd2","Type":"ContainerDied","Data":"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b"} Jul 26 18:35:21.484629 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.484510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n" event={"ID":"0c2310a8-9593-47e1-8b68-1a300ea6acd2","Type":"ContainerDied","Data":"e3af71223fbdfa4d9495e2d4aeed003825312cfcd0be6d76bffc27775b2cf9ec"} Jul 26 18:35:21.484629 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.484530 2575 scope.go:117] "RemoveContainer" containerID="03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b" Jul 26 18:35:21.493898 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.493880 2575 scope.go:117] "RemoveContainer" containerID="03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b" Jul 26 18:35:21.494198 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:35:21.494177 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b\": container with ID starting with 03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b not found: ID does not exist" containerID="03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b" Jul 26 18:35:21.494264 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.494210 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b"} err="failed to get container status \"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b\": rpc error: code = NotFound desc = could not find container \"03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b\": container with ID starting with 03b9204ec69eb856a17becc0ba9e2b7945d1045ce11cde3fa817a7b55ee5249b not found: ID does not exist" Jul 26 18:35:21.505627 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.505599 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:35:21.512029 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:21.512004 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6aa894/auth-test-model-6aa894-kserve-5fd5778479-wln9n"] Jul 26 18:35:22.210271 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:22.210237 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" path="/var/lib/kubelet/pods/0c2310a8-9593-47e1-8b68-1a300ea6acd2/volumes" Jul 26 18:35:23.104321 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.104288 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:35:23.104710 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.104697 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" containerName="main" Jul 26 18:35:23.104760 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.104711 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" containerName="main" Jul 26 18:35:23.104811 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.104782 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2310a8-9593-47e1-8b68-1a300ea6acd2" containerName="main" Jul 26 18:35:23.111627 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.111590 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.114769 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.114737 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-57f93c\"/\"openshift-service-ca.crt\"" Jul 26 18:35:23.114960 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.114742 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-57f93c\"/\"kube-root-ca.crt\"" Jul 26 18:35:23.115867 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.115818 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:35:23.116002 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.115834 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-57f93c\"/\"default-dockercfg-4rlzp\"" Jul 26 18:35:23.116072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.116047 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-57f93c\"/\"sub-test-model-57f93c-kserve-self-signed-certs\"" Jul 26 18:35:23.218041 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.217985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.218492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.218064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.218492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.218084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.218492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.218120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncllg\" (UniqueName: \"kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.218492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.218209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.218492 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.218256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.319873 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.319840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.319900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.319920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.319957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncllg\" (UniqueName: \"kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320173 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.320081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320173 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.320153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320277 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.320228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.320402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.320583 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.320564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.322234 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.322212 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.322745 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.322726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.328146 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.328124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncllg\" (UniqueName: \"kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg\") pod \"sub-test-model-57f93c-kserve-796658445-wwqt9\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.423689 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.423574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:23.563571 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:23.563512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:35:23.566117 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:35:23.566092 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f59768_917a_49ec_901e_bb09eaa30e2d.slice/crio-41fa0c930ef3b1bed43a703ece94b37a5e04c82a951451825051f1ee1d194eb0 WatchSource:0}: Error finding container 41fa0c930ef3b1bed43a703ece94b37a5e04c82a951451825051f1ee1d194eb0: Status 404 returned error can't find the container with id 41fa0c930ef3b1bed43a703ece94b37a5e04c82a951451825051f1ee1d194eb0 Jul 26 18:35:24.503913 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:24.503871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" event={"ID":"06f59768-917a-49ec-901e-bb09eaa30e2d","Type":"ContainerStarted","Data":"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a"} Jul 26 18:35:24.504316 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:24.503924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" event={"ID":"06f59768-917a-49ec-901e-bb09eaa30e2d","Type":"ContainerStarted","Data":"41fa0c930ef3b1bed43a703ece94b37a5e04c82a951451825051f1ee1d194eb0"} Jul 26 18:35:24.522405 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:24.522352 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" podStartSLOduration=1.522335829 podStartE2EDuration="1.522335829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:35:24.519862902 +0000 UTC m=+2754.918891948" watchObservedRunningTime="2026-07-26 18:35:24.522335829 +0000 UTC m=+2754.921364873" Jul 26 18:35:28.669389 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:28.669348 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:35:33.424140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:33.424097 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:33.424641 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:33.424179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:33.436745 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:33.436710 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:35:33.560481 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:35:33.560445 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:36:21.568921 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:36:21.568888 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:37:05.570778 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:05.570733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:37:22.279159 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:22.279072 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:37:33.784822 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:33.784774 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:37:37.264922 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:37.264885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:37:48.804132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:48.804089 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" podUID="06f59768-917a-49ec-901e-bb09eaa30e2d" containerName="main" containerID="cri-o://a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a" gracePeriod=45 Jul 26 18:37:49.056324 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.056256 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:37:49.100803 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100757 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.100988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.100988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100868 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.100988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100903 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncllg\" (UniqueName: \"kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.100988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100959 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.101218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.100990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm\") pod \"06f59768-917a-49ec-901e-bb09eaa30e2d\" (UID: \"06f59768-917a-49ec-901e-bb09eaa30e2d\") " Jul 26 18:37:49.101218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.101155 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home" (OuterVolumeSpecName: "home") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:37:49.101334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.101216 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:37:49.101458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.101352 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:37:49.101458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.101424 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-home\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.101458 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.101442 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-model-cache\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.103133 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.103104 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 18:37:49.103241 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.103166 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm" (OuterVolumeSpecName: "dshm") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:37:49.103241 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.103202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg" (OuterVolumeSpecName: "kube-api-access-ncllg") pod "06f59768-917a-49ec-901e-bb09eaa30e2d" (UID: "06f59768-917a-49ec-901e-bb09eaa30e2d"). InnerVolumeSpecName "kube-api-access-ncllg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:37:49.208751 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.208708 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f59768-917a-49ec-901e-bb09eaa30e2d-tls-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.208751 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.208751 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-tmp-dir\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.209004 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.208773 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncllg\" (UniqueName: \"kubernetes.io/projected/06f59768-917a-49ec-901e-bb09eaa30e2d-kube-api-access-ncllg\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.209004 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.208810 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f59768-917a-49ec-901e-bb09eaa30e2d-dshm\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:37:49.261064 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.261028 2575 generic.go:356] "Generic (PLEG): container finished" podID="06f59768-917a-49ec-901e-bb09eaa30e2d" containerID="a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a" exitCode=0 Jul 26 18:37:49.261262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.261084 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" Jul 26 18:37:49.261262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.261112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" event={"ID":"06f59768-917a-49ec-901e-bb09eaa30e2d","Type":"ContainerDied","Data":"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a"} Jul 26 18:37:49.261262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.261160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9" event={"ID":"06f59768-917a-49ec-901e-bb09eaa30e2d","Type":"ContainerDied","Data":"41fa0c930ef3b1bed43a703ece94b37a5e04c82a951451825051f1ee1d194eb0"} Jul 26 18:37:49.261262 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.261182 2575 scope.go:117] "RemoveContainer" containerID="a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a" Jul 26 18:37:49.273989 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.273970 2575 scope.go:117] "RemoveContainer" containerID="a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a" Jul 26 18:37:49.274261 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:37:49.274243 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a\": container with ID starting with a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a not found: ID does not exist" containerID="a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a" Jul 26 18:37:49.274307 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.274270 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a"} err="failed to get container status \"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a\": rpc error: code = NotFound desc = could not find container \"a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a\": container with ID starting with a1b9c3e943553311b3a995e0defe7634ffe369cfa9d7aba99bb1a6e0f8da732a not found: ID does not exist" Jul 26 18:37:49.282349 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.282326 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:37:49.288263 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:49.288241 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-57f93c/sub-test-model-57f93c-kserve-796658445-wwqt9"] Jul 26 18:37:50.209169 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:50.209135 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f59768-917a-49ec-901e-bb09eaa30e2d" path="/var/lib/kubelet/pods/06f59768-917a-49ec-901e-bb09eaa30e2d/volumes" Jul 26 18:37:51.565117 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:37:51.565078 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:38:15.761851 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:38:15.761813 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:38:53.669564 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:38:53.669483 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:39:07.859862 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:07.859828 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:39:22.177410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:22.177367 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:39:38.057984 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:38.057942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:39:47.186619 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.186576 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:39:47.187256 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.187234 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f59768-917a-49ec-901e-bb09eaa30e2d" containerName="main" Jul 26 18:39:47.187344 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.187258 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f59768-917a-49ec-901e-bb09eaa30e2d" containerName="main" Jul 26 18:39:47.187410 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.187398 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f59768-917a-49ec-901e-bb09eaa30e2d" containerName="main" Jul 26 18:39:47.190640 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.190613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.195132 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.195108 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-76b936-openshift-default-dockercfg-m67vd\"" Jul 26 18:39:47.214425 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.214395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:39:47.268966 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.268921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.268978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269499 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269499 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.269499 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.269350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggbj\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370489 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370489 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggbj\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.370740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371005 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.370770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371072 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371186 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371186 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371412 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.371774 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.371756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.373462 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.373428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.373600 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.373570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.380078 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.380048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.380176 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.380159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggbj\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj\") pod \"e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.502502 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.502411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:47.670471 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.670313 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:39:47.673376 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:39:47.673338 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84d4997_dd93_4193_b295_6ef9157cc381.slice/crio-53f3af70d92a437ab89047b0ea78879c189caf0fda84c20528e921380a227de2 WatchSource:0}: Error finding container 53f3af70d92a437ab89047b0ea78879c189caf0fda84c20528e921380a227de2: Status 404 returned error can't find the container with id 53f3af70d92a437ab89047b0ea78879c189caf0fda84c20528e921380a227de2 Jul 26 18:39:47.675675 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.675640 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:39:47.675832 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.675706 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:39:47.675832 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.675734 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:39:47.883536 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.883497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerStarted","Data":"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833"} Jul 26 18:39:47.883748 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.883555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerStarted","Data":"53f3af70d92a437ab89047b0ea78879c189caf0fda84c20528e921380a227de2"} Jul 26 18:39:47.911678 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:47.911624 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podStartSLOduration=0.911609034 podStartE2EDuration="911.609034ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:39:47.909111683 +0000 UTC m=+3018.308140730" watchObservedRunningTime="2026-07-26 18:39:47.911609034 +0000 UTC m=+3018.310638079" Jul 26 18:39:48.503446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:48.503411 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:48.508561 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:48.508535 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:48.889015 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:48.888981 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:48.889975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:48.889954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:39:59.969504 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:39:59.969468 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:40:05.442186 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.442145 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:40:05.446334 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.446308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.449137 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.449112 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3e8329-openshift-default-dockercfg-xdjd6\"" Jul 26 18:40:05.456472 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.456433 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:40:05.560050 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560251 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560251 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560382 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560440 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560440 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560557 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6fj\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560666 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.560886 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.560868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662146 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662146 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662394 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6fj\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662736 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662736 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662652 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662904 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.662999 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.662963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.663286 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.663266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.664777 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.664752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.665031 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.665012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.670750 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.670726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6fj\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.671084 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.671064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token\") pod \"e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.759740 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.759647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:05.909918 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.909839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:40:05.912500 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:40:05.912462 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719cac83_67bc_4d59_9970_a3282649baaa.slice/crio-82fe875317db544e235d32ba315dccec76e37edbd3d4515105c32a48bd353d39 WatchSource:0}: Error finding container 82fe875317db544e235d32ba315dccec76e37edbd3d4515105c32a48bd353d39: Status 404 returned error can't find the container with id 82fe875317db544e235d32ba315dccec76e37edbd3d4515105c32a48bd353d39 Jul 26 18:40:05.915471 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.915424 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:05.915594 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.915524 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:05.915594 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.915570 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:05.985506 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.985476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerStarted","Data":"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581"} Jul 26 18:40:05.985616 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:05.985534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerStarted","Data":"82fe875317db544e235d32ba315dccec76e37edbd3d4515105c32a48bd353d39"} Jul 26 18:40:07.014098 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.014035 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podStartSLOduration=2.014002511 podStartE2EDuration="2.014002511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:40:07.011343649 +0000 UTC m=+3037.410372772" watchObservedRunningTime="2026-07-26 18:40:07.014002511 +0000 UTC m=+3037.413031555" Jul 26 18:40:07.760263 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.760222 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:07.765207 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.765181 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:07.998970 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.998874 2575 generic.go:356] "Generic (PLEG): container finished" podID="f84d4997-dd93-4193-b295-6ef9157cc381" containerID="93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833" exitCode=0 Jul 26 18:40:07.998970 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.998951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerDied","Data":"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833"} Jul 26 18:40:07.999185 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.999167 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:07.999571 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:07.999556 2575 scope.go:117] "RemoveContainer" containerID="93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833" Jul 26 18:40:08.000401 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:08.000192 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:08.000605 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:08.000576 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:08.000680 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:08.000663 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:08.000746 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:08.000714 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:09.016736 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:09.016685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerStarted","Data":"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61"} Jul 26 18:40:09.502893 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:09.502859 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:40:09.503085 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:09.503033 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 18:40:09.503156 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:09.503092 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 18:40:10.503454 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:10.503415 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 18:40:10.503880 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:10.503501 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 18:40:11.502913 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:11.502872 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 18:40:11.503098 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:11.502940 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 18:40:12.503637 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:12.503602 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 26 18:40:12.504061 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:12.503670 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 26 18:40:13.506503 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:13.506463 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:40:13.507011 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:13.506817 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:40:13.507557 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:13.507539 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:40:15.267449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:15.267413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:40:23.454975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:23.453537 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:37912->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 26 18:40:23.454975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:23.453625 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:37912->10.134.0.56:15021: read: connection reset by peer" Jul 26 18:40:24.103499 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.103465 2575 generic.go:356] "Generic (PLEG): container finished" podID="719cac83-67bc-4d59-9970-a3282649baaa" containerID="4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581" exitCode=0 Jul 26 18:40:24.103680 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.103539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerDied","Data":"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581"} Jul 26 18:40:24.104043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.104028 2575 scope.go:117] "RemoveContainer" containerID="4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581" Jul 26 18:40:24.105113 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.105086 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:24.105182 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.105160 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:24.105218 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:24.105194 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 26 18:40:25.110934 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:25.110900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerStarted","Data":"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82"} Jul 26 18:40:25.759863 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:25.759822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:25.760151 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:25.760122 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 18:40:25.760220 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:25.760188 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 18:40:26.760155 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:26.760108 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 18:40:26.760647 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:26.760190 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 18:40:27.760145 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:27.760104 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 18:40:27.760363 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:27.760189 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 18:40:28.760863 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:28.760822 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 26 18:40:28.761253 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:28.760936 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 26 18:40:29.763635 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:29.763601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:29.764066 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:29.764017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:29.764596 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:29.764579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:40:48.563999 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:40:48.563966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:41:15.797463 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:15.797425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:41:29.088635 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.088598 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:29.092623 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.092601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:29.101201 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.101172 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:29.214022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.213986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspp8\" (UniqueName: \"kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8\") pod \"kuadrant-operator-catalog-jzklm\" (UID: \"826620d0-a9cb-40de-a725-37816c13abe5\") " pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:29.315370 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.315330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspp8\" (UniqueName: \"kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8\") pod \"kuadrant-operator-catalog-jzklm\" (UID: \"826620d0-a9cb-40de-a725-37816c13abe5\") " pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:29.324697 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.324664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspp8\" (UniqueName: \"kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8\") pod \"kuadrant-operator-catalog-jzklm\" (UID: \"826620d0-a9cb-40de-a725-37816c13abe5\") " pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:29.404664 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.404615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:29.547763 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.547733 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:29.549751 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:41:29.549717 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826620d0_a9cb_40de_a725_37816c13abe5.slice/crio-1af75b97344b3163f081a96474fdbda0ad22780062329bf697281d1aebe6d8df WatchSource:0}: Error finding container 1af75b97344b3163f081a96474fdbda0ad22780062329bf697281d1aebe6d8df: Status 404 returned error can't find the container with id 1af75b97344b3163f081a96474fdbda0ad22780062329bf697281d1aebe6d8df Jul 26 18:41:29.551359 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:29.551338 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 18:41:30.454197 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:30.454152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" event={"ID":"826620d0-a9cb-40de-a725-37816c13abe5","Type":"ContainerStarted","Data":"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358"} Jul 26 18:41:30.454581 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:30.454216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" event={"ID":"826620d0-a9cb-40de-a725-37816c13abe5","Type":"ContainerStarted","Data":"1af75b97344b3163f081a96474fdbda0ad22780062329bf697281d1aebe6d8df"} Jul 26 18:41:30.471243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:30.471190 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" podStartSLOduration=1.085402678 podStartE2EDuration="1.471173793s" totalImagesPullingTime="385.771115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:41:30.469573235 +0000 UTC m=+3120.868602282" watchObservedRunningTime="2026-07-26 18:41:30.471173793 +0000 UTC m=+3120.870202838" Jul 26 18:41:39.405203 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:39.405162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:39.405630 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:39.405234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:39.429022 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:39.428995 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:39.525341 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:39.525312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:39.654690 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:39.654650 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:41.515946 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:41.515891 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" podUID="826620d0-a9cb-40de-a725-37816c13abe5" containerName="registry-server" containerID="cri-o://2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358" gracePeriod=2 Jul 26 18:41:41.570594 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:41.570558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:41:41.768577 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:41.768508 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:41.955198 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:41.955167 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jspp8\" (UniqueName: \"kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8\") pod \"826620d0-a9cb-40de-a725-37816c13abe5\" (UID: \"826620d0-a9cb-40de-a725-37816c13abe5\") " Jul 26 18:41:41.957247 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:41.957223 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8" (OuterVolumeSpecName: "kube-api-access-jspp8") pod "826620d0-a9cb-40de-a725-37816c13abe5" (UID: "826620d0-a9cb-40de-a725-37816c13abe5"). InnerVolumeSpecName "kube-api-access-jspp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:41:42.056189 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.056102 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspp8\" (UniqueName: \"kubernetes.io/projected/826620d0-a9cb-40de-a725-37816c13abe5-kube-api-access-jspp8\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:41:42.522146 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.522110 2575 generic.go:356] "Generic (PLEG): container finished" podID="826620d0-a9cb-40de-a725-37816c13abe5" containerID="2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358" exitCode=0 Jul 26 18:41:42.522587 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.522156 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" Jul 26 18:41:42.522587 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.522156 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" event={"ID":"826620d0-a9cb-40de-a725-37816c13abe5","Type":"ContainerDied","Data":"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358"} Jul 26 18:41:42.522587 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.522262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jzklm" event={"ID":"826620d0-a9cb-40de-a725-37816c13abe5","Type":"ContainerDied","Data":"1af75b97344b3163f081a96474fdbda0ad22780062329bf697281d1aebe6d8df"} Jul 26 18:41:42.522587 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.522287 2575 scope.go:117] "RemoveContainer" containerID="2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358" Jul 26 18:41:42.534947 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.534925 2575 scope.go:117] "RemoveContainer" containerID="2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358" Jul 26 18:41:42.535192 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:41:42.535168 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358\": container with ID starting with 2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358 not found: ID does not exist" containerID="2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358" Jul 26 18:41:42.535256 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.535206 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358"} err="failed to get container status \"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358\": rpc error: code = NotFound desc = could not find container \"2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358\": container with ID starting with 2654b866909e1789ba06df0f69e3e3b379072c8b5f971e095043193a170d2358 not found: ID does not exist" Jul 26 18:41:42.539444 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.539421 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:42.543444 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:42.543419 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jzklm"] Jul 26 18:41:44.209101 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:41:44.209069 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826620d0-a9cb-40de-a725-37816c13abe5" path="/var/lib/kubelet/pods/826620d0-a9cb-40de-a725-37816c13abe5/volumes" Jul 26 18:42:03.974346 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:42:03.974260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:42:28.762572 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:42:28.762528 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:42:39.675939 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:42:39.675901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:43:22.526744 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:22.526655 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:43:22.527257 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:22.526910 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" containerID="cri-o://975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61" gracePeriod=30 Jul 26 18:43:27.795273 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.795212 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:43:27.931583 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931538 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggbj\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931583 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931599 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931638 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931660 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931766 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931829 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.931889 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931880 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.932194 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.931948 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy\") pod \"f84d4997-dd93-4193-b295-6ef9157cc381\" (UID: \"f84d4997-dd93-4193-b295-6ef9157cc381\") " Jul 26 18:43:27.932194 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932020 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:27.932194 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932033 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:27.932194 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932139 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:27.932392 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932347 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:27.932392 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932362 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:27.932392 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932371 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:27.932870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932847 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:43:27.932975 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.932846 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data" (OuterVolumeSpecName: "istio-data") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:27.933984 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.933951 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj" (OuterVolumeSpecName: "kube-api-access-fggbj") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "kube-api-access-fggbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:43:27.934179 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.934157 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:43:27.934590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.934460 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token" (OuterVolumeSpecName: "istio-token") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:43:27.934679 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:27.934606 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f84d4997-dd93-4193-b295-6ef9157cc381" (UID: "f84d4997-dd93-4193-b295-6ef9157cc381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:28.033423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033383 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f84d4997-dd93-4193-b295-6ef9157cc381-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.033423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033414 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.033423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033424 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f84d4997-dd93-4193-b295-6ef9157cc381-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.033423 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033431 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.033699 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033443 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggbj\" (UniqueName: \"kubernetes.io/projected/f84d4997-dd93-4193-b295-6ef9157cc381-kube-api-access-fggbj\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.033699 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.033451 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f84d4997-dd93-4193-b295-6ef9157cc381-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:28.095313 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.095227 2575 generic.go:356] "Generic (PLEG): container finished" podID="f84d4997-dd93-4193-b295-6ef9157cc381" containerID="975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61" exitCode=0 Jul 26 18:43:28.095313 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.095285 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" Jul 26 18:43:28.095313 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.095305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerDied","Data":"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61"} Jul 26 18:43:28.095521 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.095358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8" event={"ID":"f84d4997-dd93-4193-b295-6ef9157cc381","Type":"ContainerDied","Data":"53f3af70d92a437ab89047b0ea78879c189caf0fda84c20528e921380a227de2"} Jul 26 18:43:28.095521 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.095383 2575 scope.go:117] "RemoveContainer" containerID="975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61" Jul 26 18:43:28.105125 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.105106 2575 scope.go:117] "RemoveContainer" containerID="93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833" Jul 26 18:43:28.113367 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.113345 2575 scope.go:117] "RemoveContainer" containerID="975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61" Jul 26 18:43:28.113635 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:43:28.113615 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61\": container with ID starting with 975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61 not found: ID does not exist" containerID="975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61" Jul 26 18:43:28.113708 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.113643 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61"} err="failed to get container status \"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61\": rpc error: code = NotFound desc = could not find container \"975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61\": container with ID starting with 975493013be2d906ab86342ad3cbd051cad3fa285e8d1e3741de8aa43745af61 not found: ID does not exist" Jul 26 18:43:28.113708 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.113660 2575 scope.go:117] "RemoveContainer" containerID="93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833" Jul 26 18:43:28.113924 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:43:28.113908 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833\": container with ID starting with 93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833 not found: ID does not exist" containerID="93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833" Jul 26 18:43:28.113974 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.113932 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833"} err="failed to get container status \"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833\": rpc error: code = NotFound desc = could not find container \"93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833\": container with ID starting with 93fdfa8174756060033967ff61179b164e7dceee75fe88259f910e0596366833 not found: ID does not exist" Jul 26 18:43:28.117533 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.117510 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:43:28.121191 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.121168 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-76b936-openshift-default-548c58bc48-vzsn8"] Jul 26 18:43:28.209526 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:28.209487 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" path="/var/lib/kubelet/pods/f84d4997-dd93-4193-b295-6ef9157cc381/volumes" Jul 26 18:43:36.762472 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:36.762432 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:43:44.308277 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:44.308241 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:43:44.308657 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:44.308459 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" containerID="cri-o://65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82" gracePeriod=30 Jul 26 18:43:44.766477 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:44.766434 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 18:43:44.766659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:44.766509 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 18:43:49.571139 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.571067 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:43:49.651456 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651415 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651501 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651585 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651659 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651649 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651696 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651761 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp6fj\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651838 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651867 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert\") pod \"719cac83-67bc-4d59-9970-a3282649baaa\" (UID: \"719cac83-67bc-4d59-9970-a3282649baaa\") " Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651832 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:49.651930 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.651865 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:49.652247 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.652222 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.652308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.652247 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.652308 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.652268 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:43:49.652419 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.652398 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:49.652531 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.652506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:49.653746 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.653726 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:43:49.654010 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.653988 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:43:49.654163 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.654143 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj" (OuterVolumeSpecName: "kube-api-access-jp6fj") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "kube-api-access-jp6fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:43:49.654163 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.654154 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "719cac83-67bc-4d59-9970-a3282649baaa" (UID: "719cac83-67bc-4d59-9970-a3282649baaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:43:49.753008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.752964 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.752998 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.753007 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6fj\" (UniqueName: \"kubernetes.io/projected/719cac83-67bc-4d59-9970-a3282649baaa-kube-api-access-jp6fj\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753008 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.753016 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753284 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.753027 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/719cac83-67bc-4d59-9970-a3282649baaa-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753284 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.753034 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/719cac83-67bc-4d59-9970-a3282649baaa-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:49.753284 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:49.753041 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/719cac83-67bc-4d59-9970-a3282649baaa-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:43:50.211238 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.211194 2575 generic.go:356] "Generic (PLEG): container finished" podID="719cac83-67bc-4d59-9970-a3282649baaa" containerID="65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82" exitCode=0 Jul 26 18:43:50.211448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.211240 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerDied","Data":"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82"} Jul 26 18:43:50.211448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.211263 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" Jul 26 18:43:50.211448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.211308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5" event={"ID":"719cac83-67bc-4d59-9970-a3282649baaa","Type":"ContainerDied","Data":"82fe875317db544e235d32ba315dccec76e37edbd3d4515105c32a48bd353d39"} Jul 26 18:43:50.211448 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.211330 2575 scope.go:117] "RemoveContainer" containerID="65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82" Jul 26 18:43:50.223355 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.223254 2575 scope.go:117] "RemoveContainer" containerID="4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581" Jul 26 18:43:50.231741 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.231722 2575 scope.go:117] "RemoveContainer" containerID="65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82" Jul 26 18:43:50.232010 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:43:50.231993 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82\": container with ID starting with 65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82 not found: ID does not exist" containerID="65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82" Jul 26 18:43:50.232067 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.232019 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82"} err="failed to get container status \"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82\": rpc error: code = NotFound desc = could not find container \"65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82\": container with ID starting with 65ba5050bb7743d02e720d3b3d16c55a38b9cefb7e0b864d08abfa0431c5ff82 not found: ID does not exist" Jul 26 18:43:50.232067 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.232035 2575 scope.go:117] "RemoveContainer" containerID="4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581" Jul 26 18:43:50.232265 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:43:50.232250 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581\": container with ID starting with 4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581 not found: ID does not exist" containerID="4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581" Jul 26 18:43:50.232341 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.232270 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581"} err="failed to get container status \"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581\": rpc error: code = NotFound desc = could not find container \"4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581\": container with ID starting with 4f88688e90b6526909d378ae8871414f87170d185d38145fe183a4d9402ac581 not found: ID does not exist" Jul 26 18:43:50.237312 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.237288 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:43:50.242773 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:50.242745 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e8329-openshift-default-84786bfbc6-rf2p5"] Jul 26 18:43:52.209256 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:52.209223 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719cac83-67bc-4d59-9970-a3282649baaa" path="/var/lib/kubelet/pods/719cac83-67bc-4d59-9970-a3282649baaa/volumes" Jul 26 18:43:58.460822 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:43:58.460773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:44:06.143577 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:06.143538 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:44:06.144017 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:06.143744 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" containerID="cri-o://1a5a1d3be7d726cd0b5f97ff2e5b4e8a9c519ccae4a45845aedd455121d7f6e2" gracePeriod=30 Jul 26 18:44:11.324316 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.324275 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerID="1a5a1d3be7d726cd0b5f97ff2e5b4e8a9c519ccae4a45845aedd455121d7f6e2" exitCode=0 Jul 26 18:44:11.324705 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.324344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerDied","Data":"1a5a1d3be7d726cd0b5f97ff2e5b4e8a9c519ccae4a45845aedd455121d7f6e2"} Jul 26 18:44:11.324705 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.324414 2575 scope.go:117] "RemoveContainer" containerID="764c68502cbfd1774f81b0ddb553acccf4becd343ca64e8a9b8586567b2c285d" Jul 26 18:44:11.407759 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.407733 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:44:11.573601 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573508 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.573601 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.573601 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573573 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.573940 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573761 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ct9g\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.573940 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573830 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.573940 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573851 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:11.573940 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.574140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.573960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.574140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574000 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.574140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574065 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo\") pod \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\" (UID: \"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09\") " Jul 26 18:44:11.574140 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574103 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:11.574353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574278 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:11.574353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:44:11.574543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574524 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.574586 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574550 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.574586 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574566 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.574586 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574581 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.574721 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.574696 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:11.575998 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.575973 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g" (OuterVolumeSpecName: "kube-api-access-5ct9g") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "kube-api-access-5ct9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:44:11.576093 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.576023 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:44:11.576162 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.576143 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:11.576399 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.576379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" (UID: "ad0e5bf6-a3a8-4044-8597-9559ed2dfc09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:44:11.676111 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.676081 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.676111 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.676107 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.676111 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.676116 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.676341 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.676124 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ct9g\" (UniqueName: \"kubernetes.io/projected/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-kube-api-access-5ct9g\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:11.676341 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:11.676133 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:12.330581 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:12.330552 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" Jul 26 18:44:12.330581 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:12.330565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465" event={"ID":"ad0e5bf6-a3a8-4044-8597-9559ed2dfc09","Type":"ContainerDied","Data":"ae4cbf7911ab8b8b093bcabdcf1d00b9cdc174c6145562badbe9060a4761be30"} Jul 26 18:44:12.331043 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:12.330625 2575 scope.go:117] "RemoveContainer" containerID="1a5a1d3be7d726cd0b5f97ff2e5b4e8a9c519ccae4a45845aedd455121d7f6e2" Jul 26 18:44:12.357041 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:12.357009 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:44:12.362192 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:12.362165 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-57f93c-openshift-default-6dc4b596d-sp465"] Jul 26 18:44:14.209461 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:14.209424 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" path="/var/lib/kubelet/pods/ad0e5bf6-a3a8-4044-8597-9559ed2dfc09/volumes" Jul 26 18:44:19.965493 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:19.965455 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7zs9"] Jul 26 18:44:28.101061 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:28.101011 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:44:28.102091 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:28.102057 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" containerID="cri-o://da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b" gracePeriod=30 Jul 26 18:44:29.954431 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:29.954393 2575 patch_prober.go:28] interesting pod/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 18:44:29.954896 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:29.954473 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 18:44:33.365677 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.365646 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:44:33.443177 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.443083 2575 generic.go:356] "Generic (PLEG): container finished" podID="56531475-5229-48c9-994a-aa939302213a" containerID="da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b" exitCode=0 Jul 26 18:44:33.443177 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.443131 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" Jul 26 18:44:33.443177 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.443158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerDied","Data":"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b"} Jul 26 18:44:33.443424 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.443202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr" event={"ID":"56531475-5229-48c9-994a-aa939302213a","Type":"ContainerDied","Data":"158e34a2e4ae26cb08fceb5ac3d50164ad04f7594e6ae850176e51d51541d927"} Jul 26 18:44:33.443424 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.443228 2575 scope.go:117] "RemoveContainer" containerID="da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b" Jul 26 18:44:33.451980 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.451959 2575 scope.go:117] "RemoveContainer" containerID="f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784" Jul 26 18:44:33.460217 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.460197 2575 scope.go:117] "RemoveContainer" containerID="da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b" Jul 26 18:44:33.460484 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:44:33.460466 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b\": container with ID starting with da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b not found: ID does not exist" containerID="da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b" Jul 26 18:44:33.460526 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.460491 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b"} err="failed to get container status \"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b\": rpc error: code = NotFound desc = could not find container \"da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b\": container with ID starting with da450321b2a365990a7fe88808dbdcd011d8738e078914a273650bb0b8bfdd7b not found: ID does not exist" Jul 26 18:44:33.460526 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.460510 2575 scope.go:117] "RemoveContainer" containerID="f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784" Jul 26 18:44:33.460751 ip-10-0-138-219 kubenswrapper[2575]: E0726 18:44:33.460731 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784\": container with ID starting with f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784 not found: ID does not exist" containerID="f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784" Jul 26 18:44:33.460817 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.460757 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784"} err="failed to get container status \"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784\": rpc error: code = NotFound desc = could not find container \"f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784\": container with ID starting with f9a7e621a4ed4894445177ef7f0f60596c81b6c39639d080ef8cb100152cc784 not found: ID does not exist" Jul 26 18:44:33.487778 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487741 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.487967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.487967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487849 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.487967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.487967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.487967 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487945 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kx46\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.488243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.487979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.488243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488015 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.488243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488080 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token\") pod \"56531475-5229-48c9-994a-aa939302213a\" (UID: \"56531475-5229-48c9-994a-aa939302213a\") " Jul 26 18:44:33.488243 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488204 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:33.488449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488286 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 18:44:33.488449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:33.488449 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488334 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:33.488599 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488538 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56531475-5229-48c9-994a-aa939302213a-istiod-ca-cert\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.488599 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488559 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-certs\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.488599 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488573 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-credential-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.488599 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.488587 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-workload-socket\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.489303 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.489275 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data" (OuterVolumeSpecName: "istio-data") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:33.490253 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.490228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 18:44:33.490353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.490316 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token" (OuterVolumeSpecName: "istio-token") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:44:33.490353 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.490322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 18:44:33.490443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.490342 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46" (OuterVolumeSpecName: "kube-api-access-9kx46") pod "56531475-5229-48c9-994a-aa939302213a" (UID: "56531475-5229-48c9-994a-aa939302213a"). InnerVolumeSpecName "kube-api-access-9kx46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 18:44:33.589446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.589406 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-istio-token\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.589446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.589438 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56531475-5229-48c9-994a-aa939302213a-istio-podinfo\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.589446 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.589450 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-envoy\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.589680 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.589462 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kx46\" (UniqueName: \"kubernetes.io/projected/56531475-5229-48c9-994a-aa939302213a-kube-api-access-9kx46\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.589680 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.589475 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56531475-5229-48c9-994a-aa939302213a-istio-data\") on node \"ip-10-0-138-219.ec2.internal\" DevicePath \"\"" Jul 26 18:44:33.768265 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.768229 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:44:33.771973 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:33.771946 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6aa894-openshift-default-89d57cf88-v8ldr"] Jul 26 18:44:34.149258 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:34.149221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ppz9v_94e3d136-7f69-457b-82ea-1264ec5c2d1f/manager/0.log" Jul 26 18:44:34.209050 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:34.209019 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56531475-5229-48c9-994a-aa939302213a" path="/var/lib/kubelet/pods/56531475-5229-48c9-994a-aa939302213a/volumes" Jul 26 18:44:34.268346 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:34.268308 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8r2wz_158ef9f0-f1f5-4345-b527-5790520d5b04/manager/0.log" Jul 26 18:44:34.974761 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:34.974730 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b79b7df6c-v7h2v_476a8e69-43a2-4297-b3c5-61bc079361f4/copy-manifests/0.log" Jul 26 18:44:34.995013 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:34.994976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b79b7df6c-v7h2v_476a8e69-43a2-4297-b3c5-61bc079361f4/manager/0.log" Jul 26 18:44:37.285198 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:37.285163 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hbblt_00f88d38-30ba-4904-a23e-a47e6c85822e/manager/0.log" Jul 26 18:44:37.513251 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:37.513209 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6w7ls_9bd37f86-de00-4c07-b85d-3dec6de95d5c/registry-server/0.log" Jul 26 18:44:37.748241 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:37.748207 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q7zs9_bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f/limitador/0.log" Jul 26 18:44:37.870450 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:37.870399 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r5gfn_fff2f629-14af-4d71-bc14-81a03f68682d/manager/0.log" Jul 26 18:44:38.220838 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:38.220806 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb_8299eeda-2431-4c63-a25a-b0ef54e83123/istio-proxy/0.log" Jul 26 18:44:38.690261 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:38.690202 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/1.log" Jul 26 18:44:39.381612 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:39.381580 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fv9pg_72b9335a-d820-4dfa-b2a2-821ae96daaed/main/0.log" Jul 26 18:44:39.944408 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:39.944376 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5s462_bdf0636d-9254-4997-a939-6ee6a74b27d0/main/0.log" Jul 26 18:44:48.462890 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:48.462857 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6dpt6_eb57ba8d-0081-4e01-baec-552dfcc34995/global-pull-secret-syncer/0.log" Jul 26 18:44:48.656294 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:48.656256 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g9tw7_8e879f3c-533b-44ca-bb2e-aafe77e9c9d1/konnectivity-agent/0.log" Jul 26 18:44:48.779190 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:48.779075 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-219.ec2.internal_1de7be92a0a914801205d3ece54cca0e/haproxy/0.log" Jul 26 18:44:54.225088 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:54.225058 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hbblt_00f88d38-30ba-4904-a23e-a47e6c85822e/manager/0.log" Jul 26 18:44:54.305317 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:54.305283 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6w7ls_9bd37f86-de00-4c07-b85d-3dec6de95d5c/registry-server/0.log" Jul 26 18:44:54.542242 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:54.542147 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q7zs9_bfc0bd0a-2cf4-4d6f-80fb-534dfb20b55f/limitador/0.log" Jul 26 18:44:54.647145 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:54.647106 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r5gfn_fff2f629-14af-4d71-bc14-81a03f68682d/manager/0.log" Jul 26 18:44:56.413083 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.413052 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/alertmanager/0.log" Jul 26 18:44:56.438445 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.438410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/config-reloader/0.log" Jul 26 18:44:56.459379 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.459347 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/kube-rbac-proxy-web/0.log" Jul 26 18:44:56.483746 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.483671 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/kube-rbac-proxy/0.log" Jul 26 18:44:56.504801 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.504761 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/kube-rbac-proxy-metric/0.log" Jul 26 18:44:56.525753 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.525727 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/prom-label-proxy/0.log" Jul 26 18:44:56.548505 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.548478 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2109ce76-8557-4b0b-b98e-7cd3602ff608/init-config-reloader/0.log" Jul 26 18:44:56.628370 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.628333 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-66zt2_6aaec6c9-157f-46e2-88b2-9e496bf29cb3/kube-state-metrics/0.log" Jul 26 18:44:56.643199 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.643173 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-66zt2_6aaec6c9-157f-46e2-88b2-9e496bf29cb3/kube-rbac-proxy-main/0.log" Jul 26 18:44:56.662098 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.662068 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-66zt2_6aaec6c9-157f-46e2-88b2-9e496bf29cb3/kube-rbac-proxy-self/0.log" Jul 26 18:44:56.686755 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.686721 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c755fcd9f-j4pdq_8547f091-de47-4e0d-a516-f058c820a624/metrics-server/0.log" Jul 26 18:44:56.710985 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.710961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-h6vd7_05cc1b09-4aca-4e83-9a7a-de0063b9b204/monitoring-plugin/0.log" Jul 26 18:44:56.805580 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.805505 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rjqfq_4ce6769a-307c-419e-aa93-95148a37983a/node-exporter/0.log" Jul 26 18:44:56.824292 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.824264 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rjqfq_4ce6769a-307c-419e-aa93-95148a37983a/kube-rbac-proxy/0.log" Jul 26 18:44:56.842605 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.842577 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rjqfq_4ce6769a-307c-419e-aa93-95148a37983a/init-textfile/0.log" Jul 26 18:44:56.944941 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.944910 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5f9l8_21ab5303-328c-4cde-97fc-8501d82c0a41/kube-rbac-proxy-main/0.log" Jul 26 18:44:56.964731 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.964694 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5f9l8_21ab5303-328c-4cde-97fc-8501d82c0a41/kube-rbac-proxy-self/0.log" Jul 26 18:44:56.985549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:56.985521 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5f9l8_21ab5303-328c-4cde-97fc-8501d82c0a41/openshift-state-metrics/0.log" Jul 26 18:44:57.021521 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.021492 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/prometheus/0.log" Jul 26 18:44:57.039916 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.039888 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/config-reloader/0.log" Jul 26 18:44:57.060231 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.060149 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/thanos-sidecar/0.log" Jul 26 18:44:57.079493 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.079459 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/kube-rbac-proxy-web/0.log" Jul 26 18:44:57.102080 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.102056 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/kube-rbac-proxy/0.log" Jul 26 18:44:57.122017 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.121991 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/kube-rbac-proxy-thanos/0.log" Jul 26 18:44:57.141957 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.141931 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3001111b-f282-4266-8c6a-84079ecf8555/init-config-reloader/0.log" Jul 26 18:44:57.172106 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.172077 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hxh2g_910330eb-d33c-452b-bfca-fc1730835f71/prometheus-operator/0.log" Jul 26 18:44:57.187497 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.187471 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-hxh2g_910330eb-d33c-452b-bfca-fc1730835f71/kube-rbac-proxy/0.log" Jul 26 18:44:57.210260 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.210229 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-m9r4k_adadb65e-6276-4c4f-b603-824019664192/prometheus-operator-admission-webhook/0.log" Jul 26 18:44:57.314858 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.314758 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/thanos-query/0.log" Jul 26 18:44:57.334593 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.334570 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/kube-rbac-proxy-web/0.log" Jul 26 18:44:57.355841 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.355815 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/kube-rbac-proxy/0.log" Jul 26 18:44:57.376830 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.376782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/prom-label-proxy/0.log" Jul 26 18:44:57.398250 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.398214 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/kube-rbac-proxy-rules/0.log" Jul 26 18:44:57.422161 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:57.422123 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fdcffdbfd-lnbmb_012e7260-d0d1-403f-87c8-a0e3570e718e/kube-rbac-proxy-metrics/0.log" Jul 26 18:44:58.313301 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313264 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75"] Jul 26 18:44:58.313754 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313733 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313756 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313772 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313780 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313810 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313819 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313842 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313851 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313860 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826620d0-a9cb-40de-a725-37816c13abe5" containerName="registry-server" Jul 26 18:44:58.313870 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313868 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="826620d0-a9cb-40de-a725-37816c13abe5" containerName="registry-server" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313888 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313901 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.313910 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314018 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314032 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0e5bf6-a3a8-4044-8597-9559ed2dfc09" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314042 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="719cac83-67bc-4d59-9970-a3282649baaa" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314051 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314061 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="826620d0-a9cb-40de-a725-37816c13abe5" containerName="registry-server" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314071 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314080 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84d4997-dd93-4193-b295-6ef9157cc381" containerName="istio-proxy" Jul 26 18:44:58.314543 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.314089 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56531475-5229-48c9-994a-aa939302213a" containerName="istio-proxy" Jul 26 18:44:58.317642 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.317622 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.320081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.320061 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wprln\"/\"openshift-service-ca.crt\"" Jul 26 18:44:58.320174 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.320099 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wprln\"/\"kube-root-ca.crt\"" Jul 26 18:44:58.321269 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.321252 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wprln\"/\"default-dockercfg-6tfqw\"" Jul 26 18:44:58.326549 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.326529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75"] Jul 26 18:44:58.417814 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.417751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9nb\" (UniqueName: \"kubernetes.io/projected/ad427edd-30d7-4d5a-b84f-c358571a8d57-kube-api-access-hx9nb\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.417986 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.417871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-lib-modules\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.417986 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.417904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-podres\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.417986 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.417942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-sys\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.417986 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.417973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-proc\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.518656 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9nb\" (UniqueName: \"kubernetes.io/projected/ad427edd-30d7-4d5a-b84f-c358571a8d57-kube-api-access-hx9nb\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-lib-modules\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-podres\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-sys\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-lib-modules\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-podres\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-sys\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.518978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-proc\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.519099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.519061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ad427edd-30d7-4d5a-b84f-c358571a8d57-proc\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.526273 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.526248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9nb\" (UniqueName: \"kubernetes.io/projected/ad427edd-30d7-4d5a-b84f-c358571a8d57-kube-api-access-hx9nb\") pod \"perf-node-gather-daemonset-fzp75\" (UID: \"ad427edd-30d7-4d5a-b84f-c358571a8d57\") " pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.628512 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.628484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:58.767871 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:58.767845 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75"] Jul 26 18:44:58.769751 ip-10-0-138-219 kubenswrapper[2575]: W0726 18:44:58.769722 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podad427edd_30d7_4d5a_b84f_c358571a8d57.slice/crio-75c085e1f95f7e913e261fbbf7391967ddc97a4461961213e4763546e6166f1f WatchSource:0}: Error finding container 75c085e1f95f7e913e261fbbf7391967ddc97a4461961213e4763546e6166f1f: Status 404 returned error can't find the container with id 75c085e1f95f7e913e261fbbf7391967ddc97a4461961213e4763546e6166f1f Jul 26 18:44:59.581517 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:59.581482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" event={"ID":"ad427edd-30d7-4d5a-b84f-c358571a8d57","Type":"ContainerStarted","Data":"e97fbefa5ff326b97fb117e7196dc1db051b166ba7ec8763218bb4bd12641289"} Jul 26 18:44:59.582081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:59.581546 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" event={"ID":"ad427edd-30d7-4d5a-b84f-c358571a8d57","Type":"ContainerStarted","Data":"75c085e1f95f7e913e261fbbf7391967ddc97a4461961213e4763546e6166f1f"} Jul 26 18:44:59.582081 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:59.581575 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:44:59.598099 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:44:59.598043 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" podStartSLOduration=1.598028098 podStartE2EDuration="1.598028098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 18:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 18:44:59.596647273 +0000 UTC m=+3329.995676316" watchObservedRunningTime="2026-07-26 18:44:59.598028098 +0000 UTC m=+3329.997057144" Jul 26 18:45:01.384415 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:01.384387 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9jrk_07f0a26c-bdb1-4170-aea6-884aadf1345c/dns/0.log" Jul 26 18:45:01.407275 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:01.407243 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g9jrk_07f0a26c-bdb1-4170-aea6-884aadf1345c/kube-rbac-proxy/0.log" Jul 26 18:45:01.485988 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:01.485960 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fqjvs_223047b2-a308-42a1-b28c-5be77c548d7a/dns-node-resolver/0.log" Jul 26 18:45:02.103380 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:02.103346 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dbvcm_430af05c-ecbe-4796-b18b-8c11872a3d84/node-ca/0.log" Jul 26 18:45:03.058185 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:03.058145 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb_8299eeda-2431-4c63-a25a-b0ef54e83123/istio-proxy/0.log" Jul 26 18:45:03.406360 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:03.406325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/0.log" Jul 26 18:45:03.461185 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:03.461143 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/1.log" Jul 26 18:45:04.442047 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:04.442002 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rg2p5_a8709f3b-1af2-4562-946a-19504964c505/serve-healthcheck-canary/0.log" Jul 26 18:45:05.024568 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:05.024540 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5cc5_4c284f4f-37ab-4491-9099-73320fd0c7a0/kube-rbac-proxy/0.log" Jul 26 18:45:05.045964 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:05.045939 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5cc5_4c284f4f-37ab-4491-9099-73320fd0c7a0/exporter/0.log" Jul 26 18:45:05.069377 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:05.069338 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5cc5_4c284f4f-37ab-4491-9099-73320fd0c7a0/extractor/0.log" Jul 26 18:45:05.597160 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:05.597135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wprln/perf-node-gather-daemonset-fzp75" Jul 26 18:45:07.442391 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:07.442345 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ppz9v_94e3d136-7f69-457b-82ea-1264ec5c2d1f/manager/0.log" Jul 26 18:45:07.832973 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:07.832942 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8r2wz_158ef9f0-f1f5-4345-b527-5790520d5b04/manager/0.log" Jul 26 18:45:08.450003 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:08.449956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b79b7df6c-v7h2v_476a8e69-43a2-4297-b3c5-61bc079361f4/manager/0.log" Jul 26 18:45:08.468590 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:08.468558 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b79b7df6c-v7h2v_476a8e69-43a2-4297-b3c5-61bc079361f4/copy-manifests/0.log" Jul 26 18:45:08.851443 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:08.851408 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb_8299eeda-2431-4c63-a25a-b0ef54e83123/istio-proxy/0.log" Jul 26 18:45:09.153293 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:09.153257 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/0.log" Jul 26 18:45:09.204655 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:09.204616 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/1.log" Jul 26 18:45:09.640119 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:09.640086 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fdddb58cf-t946x_de976e4e-544b-45ac-824d-7378b74741ea/manager/0.log" Jul 26 18:45:10.652067 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:10.652040 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cmk2rb_8299eeda-2431-4c63-a25a-b0ef54e83123/istio-proxy/0.log" Jul 26 18:45:10.967464 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:10.967381 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/0.log" Jul 26 18:45:11.020025 ip-10-0-138-219 kubenswrapper[2575]: I0726 18:45:11.019996 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-9nbc2_cec9cd99-7139-47b9-8aaf-f2ad972f6809/istio-proxy/1.log"