Aug 05 17:31:03.062181 ip-10-0-143-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 05 17:31:03.062193 ip-10-0-143-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 05 17:31:03.062199 ip-10-0-143-87 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 05 17:31:03.062415 ip-10-0-143-87 systemd[1]: Failed to start Kubernetes Kubelet. Aug 05 17:31:13.163438 ip-10-0-143-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 05 17:31:13.163454 ip-10-0-143-87 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e67675c084c5432ab8544b7cf491760d -- Aug 05 17:32:53.398556 ip-10-0-143-87 systemd[1]: Starting Kubernetes Kubelet... Aug 05 17:32:53.831175 ip-10-0-143-87 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:32:53.831175 ip-10-0-143-87 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 05 17:32:53.831175 ip-10-0-143-87 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:32:53.831175 ip-10-0-143-87 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 05 17:32:53.831175 ip-10-0-143-87 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 05 17:32:53.832777 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.832661 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 05 17:32:53.838799 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838784 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838802 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838806 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838810 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838815 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838819 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838823 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838826 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838829 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838832 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838835 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838838 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.838836 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838841 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838844 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838846 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838849 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838851 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838854 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838857 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838860 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838862 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838872 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838875 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838878 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838880 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838882 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838885 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838888 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838890 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838892 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838896 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.839120 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838899 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838902 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838906 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838910 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838913 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838916 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838919 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838922 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838925 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838927 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838930 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838933 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838935 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838938 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838941 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838944 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838947 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838950 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838952 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838955 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.839626 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838958 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838960 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838963 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838965 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838967 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838970 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838973 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838976 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838979 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838981 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838984 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838986 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838988 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838993 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838995 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.838998 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839001 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839003 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839005 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.840101 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839008 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839011 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839013 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839016 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839018 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839020 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839023 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839026 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839029 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839031 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839034 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839037 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839040 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839042 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839044 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839047 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839049 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839052 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839055 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839057 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.840567 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839060 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839062 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839064 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839067 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839069 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839072 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839074 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839077 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839079 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839827 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839834 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839837 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839839 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839842 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839845 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839847 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839850 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839853 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839855 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839858 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.841042 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839861 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839864 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839867 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839870 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839873 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839876 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839878 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839881 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839883 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839885 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839888 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839891 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839894 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839896 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839899 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839901 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839904 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839906 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839909 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839911 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.841542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839914 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839916 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839919 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839921 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839923 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839926 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839928 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839931 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839933 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839936 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839939 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839942 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839945 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839948 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839950 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839953 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839956 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839959 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839962 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839964 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.842039 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839967 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839978 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839981 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839983 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839986 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839988 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839991 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839994 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839996 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.839999 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840002 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840006 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840010 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840013 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840016 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840019 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840021 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840024 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840026 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.842542 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840029 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840031 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840034 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840037 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840040 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840043 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840048 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840052 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840054 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840057 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840061 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840063 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840066 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840068 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840071 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840073 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840076 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840078 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840081 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.843029 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840083 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840085 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840088 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840091 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840093 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840095 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840098 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840101 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840106 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840109 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840496 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840501 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840504 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840507 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840509 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840512 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840515 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840520 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840524 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840528 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.843523 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840531 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840534 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840537 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840540 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840543 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840545 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840548 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840551 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840554 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840557 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840559 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840562 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840565 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840568 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840571 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840574 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840577 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840579 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840582 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.843995 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840585 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840588 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840590 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840593 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840596 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840598 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840601 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840603 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840606 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840608 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840612 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840616 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840618 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840621 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840624 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840627 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840630 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840633 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840636 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.844497 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840639 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840641 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840644 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840647 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840650 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840652 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840655 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840657 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840660 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840662 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840665 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840668 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840670 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840673 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840676 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840678 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840681 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840684 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840686 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840689 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.844951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840691 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840693 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840696 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840698 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840701 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840703 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840706 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840708 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840711 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840713 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840716 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840719 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840721 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840724 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840728 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840730 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840733 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840735 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840738 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840740 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.845439 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840742 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840745 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840747 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840751 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840755 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840757 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840760 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840762 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840764 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840767 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.840769 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840857 2561 flags.go:64] FLAG: --address="0.0.0.0" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840875 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840886 2561 flags.go:64] FLAG: --anonymous-auth="true" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840894 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840899 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840903 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840907 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840911 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840914 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840917 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 05 17:32:53.845937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840921 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840924 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840927 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840932 2561 flags.go:64] FLAG: --cgroup-root="" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840935 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840938 2561 flags.go:64] FLAG: --client-ca-file="" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840941 2561 flags.go:64] FLAG: --cloud-provider="external" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840944 2561 flags.go:64] FLAG: --cluster-dns="[]" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840951 2561 flags.go:64] FLAG: --cluster-domain="" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840954 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840957 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840960 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840963 2561 flags.go:64] FLAG: --container-log-max-files="5" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840967 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840970 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840973 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840977 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840980 2561 flags.go:64] FLAG: --contention-profiling="false" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840982 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840985 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840988 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840991 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840996 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.840999 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841002 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 05 17:32:53.846447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841004 2561 flags.go:64] FLAG: --enable-load-reader="false" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841007 2561 flags.go:64] FLAG: --enable-server="true" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841010 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841015 2561 flags.go:64] FLAG: --event-burst="100" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841018 2561 flags.go:64] FLAG: --event-qps="50" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841020 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841024 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841027 2561 flags.go:64] FLAG: --eviction-hard="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841030 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841033 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841036 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841039 2561 flags.go:64] FLAG: --eviction-soft="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841042 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841045 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841048 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841051 2561 flags.go:64] FLAG: --experimental-mounter-path="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841054 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841057 2561 flags.go:64] FLAG: --fail-swap-on="true" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841060 2561 flags.go:64] FLAG: --feature-gates="" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841063 2561 flags.go:64] FLAG: --file-check-frequency="20s" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841066 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841069 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841072 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841075 2561 flags.go:64] FLAG: --healthz-port="10248" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841078 2561 flags.go:64] FLAG: --help="false" Aug 05 17:32:53.847025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841081 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-143-87.ec2.internal" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841084 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841087 2561 flags.go:64] FLAG: --http-check-frequency="20s" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841089 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841093 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841096 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841099 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841101 2561 flags.go:64] FLAG: --image-service-endpoint="" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841104 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841107 2561 flags.go:64] FLAG: --kube-api-burst="100" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841110 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841113 2561 flags.go:64] FLAG: --kube-api-qps="50" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841116 2561 flags.go:64] FLAG: --kube-reserved="" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841119 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841122 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841125 2561 flags.go:64] FLAG: --kubelet-cgroups="" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841127 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841130 2561 flags.go:64] FLAG: --lock-file="" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841133 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841136 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841139 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841145 2561 flags.go:64] FLAG: --log-json-split-stream="false" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841148 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841150 2561 flags.go:64] FLAG: --log-text-split-stream="false" Aug 05 17:32:53.847676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841153 2561 flags.go:64] FLAG: --logging-format="text" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841156 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841159 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841162 2561 flags.go:64] FLAG: --manifest-url="" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841164 2561 flags.go:64] FLAG: --manifest-url-header="" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841169 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841172 2561 flags.go:64] FLAG: --max-open-files="1000000" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841175 2561 flags.go:64] FLAG: --max-pods="110" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841178 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841181 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841184 2561 flags.go:64] FLAG: --memory-manager-policy="None" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841187 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841189 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841192 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841195 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841202 2561 flags.go:64] FLAG: --node-status-max-images="50" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841204 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841207 2561 flags.go:64] FLAG: --oom-score-adj="-999" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841211 2561 flags.go:64] FLAG: --pod-cidr="" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841213 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841236 2561 flags.go:64] FLAG: --pod-manifest-path="" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841239 2561 flags.go:64] FLAG: --pod-max-pids="-1" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841243 2561 flags.go:64] FLAG: --pods-per-core="0" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841246 2561 flags.go:64] FLAG: --port="10250" Aug 05 17:32:53.848266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841249 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841252 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c88afcc45bbb0aa6" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841255 2561 flags.go:64] FLAG: --qos-reserved="" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841258 2561 flags.go:64] FLAG: --read-only-port="10255" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841261 2561 flags.go:64] FLAG: --register-node="true" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841264 2561 flags.go:64] FLAG: --register-with-taints="" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841268 2561 flags.go:64] FLAG: --registry-burst="10" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841271 2561 flags.go:64] FLAG: --registry-qps="5" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841274 2561 flags.go:64] FLAG: --reserved-cpus="" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841276 2561 flags.go:64] FLAG: --reserved-memory="" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841280 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841283 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841286 2561 flags.go:64] FLAG: --rotate-certificates="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841289 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841291 2561 flags.go:64] FLAG: --runonce="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841294 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841297 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841300 2561 flags.go:64] FLAG: --seccomp-default="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841303 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841306 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841309 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841312 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841315 2561 flags.go:64] FLAG: --storage-driver-password="root" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841318 2561 flags.go:64] FLAG: --storage-driver-secure="false" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841321 2561 flags.go:64] FLAG: --storage-driver-table="stats" Aug 05 17:32:53.848831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841324 2561 flags.go:64] FLAG: --storage-driver-user="root" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841327 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841330 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841334 2561 flags.go:64] FLAG: --system-cgroups="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841337 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841342 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841345 2561 flags.go:64] FLAG: --tls-cert-file="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841348 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841352 2561 flags.go:64] FLAG: --tls-min-version="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841355 2561 flags.go:64] FLAG: --tls-private-key-file="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841358 2561 flags.go:64] FLAG: --topology-manager-policy="none" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841361 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841364 2561 flags.go:64] FLAG: --topology-manager-scope="container" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841367 2561 flags.go:64] FLAG: --v="2" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841371 2561 flags.go:64] FLAG: --version="false" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841375 2561 flags.go:64] FLAG: --vmodule="" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841379 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.841382 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841496 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841500 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841504 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841507 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841511 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.849482 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841513 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841516 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841518 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841521 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841523 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841526 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841528 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841531 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841534 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841536 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841539 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841541 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841545 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841548 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841551 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841553 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841556 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841558 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841561 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841563 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.850130 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841566 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841568 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841571 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841575 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841579 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841582 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841584 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841587 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841591 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841594 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841597 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841602 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841605 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841608 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841610 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841613 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841616 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841619 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841621 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.850631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841624 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841626 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841629 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841631 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841633 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841637 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841639 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841642 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841644 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841646 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841649 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841651 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841654 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841656 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841659 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841661 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841665 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841667 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841670 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841672 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.851084 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841674 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841677 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841679 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841682 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841685 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841688 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841690 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841692 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841695 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841698 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841700 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841702 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841705 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841707 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841710 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841712 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841715 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841718 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841721 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841723 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.851597 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841725 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841728 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841731 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841735 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841737 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841740 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841743 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841745 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841748 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841750 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841753 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841756 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841758 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841761 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.841763 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.852091 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.842394 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.849062 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.849079 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.849971 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.849979 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850037 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850042 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850046 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850049 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850054 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850058 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850061 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850064 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850067 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850070 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850073 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850075 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850078 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850080 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850083 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.852472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850086 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850089 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850091 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850094 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850096 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850099 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850102 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850104 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850107 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850110 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850112 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850114 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850117 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850120 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850124 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850126 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850129 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850131 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850134 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.852958 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850137 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850140 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850143 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850145 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850148 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850150 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850153 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850155 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850158 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850160 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850163 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850167 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850170 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850173 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850176 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850178 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850181 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850184 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850186 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850189 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.853425 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850192 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850194 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850197 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850199 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850203 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850206 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850209 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850214 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850217 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850220 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850223 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850242 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850245 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850248 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850250 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850252 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850255 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850257 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850260 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.853897 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850263 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850265 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850268 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850270 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850273 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850275 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850277 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850280 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850283 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850286 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850288 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850291 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850294 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850297 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850299 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850302 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850304 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850307 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850310 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850312 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.854368 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850315 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850318 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850322 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850324 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850327 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850329 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.850334 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850449 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850454 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850457 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850459 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850463 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850465 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850468 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 05 17:32:53.854847 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850471 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850474 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850476 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850479 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850482 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850484 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850487 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850489 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850492 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850494 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850496 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850499 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850501 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850504 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850506 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850509 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850511 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850514 2561 feature_gate.go:328] unrecognized feature gate: Example2 Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850517 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850519 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 05 17:32:53.855186 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850521 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850524 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850527 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850529 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850532 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850534 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850537 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850539 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850541 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850544 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850546 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850549 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850551 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850553 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850555 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850558 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850560 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850563 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850565 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850567 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 05 17:32:53.855691 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850570 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850572 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850574 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850577 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850581 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850585 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850588 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850590 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850593 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850596 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850599 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850601 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850604 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850607 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850611 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850614 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850617 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850619 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850622 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 05 17:32:53.856167 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850624 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850626 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850629 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850632 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850634 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850637 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850639 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850642 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850645 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850647 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850650 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850653 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850656 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850660 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850662 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850665 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850668 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850671 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850673 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 05 17:32:53.856631 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850676 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850678 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850680 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850683 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850686 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850688 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850691 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850693 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850696 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850698 2561 feature_gate.go:328] unrecognized feature gate: Example Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850701 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850704 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850706 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:53.850708 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.850713 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 05 17:32:53.857096 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.850838 2561 server.go:962] "Client rotation is on, will bootstrap in background" Aug 05 17:32:53.857554 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.853383 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 05 17:32:53.857554 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.854743 2561 server.go:1019] "Starting client certificate rotation" Aug 05 17:32:53.857554 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.854838 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:32:53.857554 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.855748 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 05 17:32:53.857554 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.857515 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:32:53.864279 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.864252 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 05 17:32:53.876331 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.876312 2561 log.go:25] "Validated CRI v1 runtime API" Aug 05 17:32:53.878312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.878297 2561 log.go:25] "Validated CRI v1 image API" Aug 05 17:32:53.879106 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.879085 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 05 17:32:53.883547 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.883527 2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a797e36c-688b-42b6-849b-744e91a416fb:/dev/nvme0n1p3 f8dceae8-92c6-4655-976a-091d7ce1fd85:/dev/nvme0n1p4] Aug 05 17:32:53.883623 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.883545 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 05 17:32:53.884586 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.884571 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:32:53.888833 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.888727 2561 manager.go:216] Machine: {Timestamp:2026-08-05 17:32:53.887417078 +0000 UTC m=+0.363699535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111372 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e73e357cfc5c403778a91971d3137 SystemUUID:ec2e73e3-57cf-c5c4-0377-8a91971d3137 BootID:e67675c0-84c5-432a-b854-4b7cf491760d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:14:d4:d9:a2:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:14:d4:d9:a2:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:74:62:e5:e6:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 05 17:32:53.888833 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.888824 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 05 17:32:53.888960 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.888906 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 05 17:32:53.892041 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892018 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 05 17:32:53.892181 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892044 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 05 17:32:53.892873 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892863 2561 topology_manager.go:138] "Creating topology manager with none policy" Aug 05 17:32:53.892909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892876 2561 container_manager_linux.go:309] "Creating device plugin manager" Aug 05 17:32:53.892909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892883 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:32:53.892909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.892897 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 05 17:32:53.893710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.893700 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 05 17:32:53.896971 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.896959 2561 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:32:53.897088 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897078 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 05 17:32:53.897129 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897108 2561 kubelet.go:486] "Attempting to sync node with API server" Aug 05 17:32:53.897129 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897117 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 05 17:32:53.897199 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897137 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 05 17:32:53.897199 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897145 2561 kubelet.go:398] "Adding apiserver pod source" Aug 05 17:32:53.897199 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897158 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 05 17:32:53.897530 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897516 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:32:53.897577 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.897545 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 05 17:32:53.899936 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.899920 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 05 17:32:53.901218 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.901203 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 05 17:32:53.901299 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.901249 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 05 17:32:53.902521 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902509 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902525 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902531 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902537 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902545 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902550 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902559 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902564 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 05 17:32:53.902571 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902570 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 05 17:32:53.902794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902579 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 05 17:32:53.902794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902591 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 05 17:32:53.902794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.902600 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 05 17:32:53.904375 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.904351 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 05 17:32:53.904489 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.904476 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 05 17:32:53.908082 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.908064 2561 server.go:1268] "Started kubelet" Aug 05 17:32:53.908168 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.908129 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 05 17:32:53.908290 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.908198 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 05 17:32:53.908334 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.908314 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 05 17:32:53.908886 ip-10-0-143-87 systemd[1]: Started Kubernetes Kubelet. Aug 05 17:32:53.909193 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.909176 2561 server.go:310] "Adding debug handlers to kubelet server" Aug 05 17:32:53.909451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.909433 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 05 17:32:53.909839 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.909820 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 05 17:32:53.909982 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.909918 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 05 17:32:53.909982 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.909944 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 05 17:32:53.914415 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.913369 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-87.ec2.internal.18c8f9215afadeef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-87.ec2.internal,UID:ip-10-0-143-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-87.ec2.internal,},FirstTimestamp:2026-08-05 17:32:53.908029167 +0000 UTC m=+0.384311623,LastTimestamp:2026-08-05 17:32:53.908029167 +0000 UTC m=+0.384311623,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-87.ec2.internal,}" Aug 05 17:32:53.914786 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.914769 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 05 17:32:53.915358 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.915338 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 05 17:32:53.915443 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.915341 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 05 17:32:53.915443 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.915436 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:53.916150 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916135 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 05 17:32:53.916150 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916151 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 05 17:32:53.916301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916136 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 05 17:32:53.916301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916256 2561 reconstruct.go:97] "Volume reconstruction finished" Aug 05 17:32:53.916301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916266 2561 reconciler.go:29] "Reconciler: start to sync state" Aug 05 17:32:53.916427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916360 2561 factory.go:153] Registering CRI-O factory Aug 05 17:32:53.916427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916375 2561 factory.go:223] Registration of the crio container factory successfully Aug 05 17:32:53.916427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916420 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 05 17:32:53.916427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916427 2561 factory.go:55] Registering systemd factory Aug 05 17:32:53.916593 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916433 2561 factory.go:223] Registration of the systemd container factory successfully Aug 05 17:32:53.916593 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916454 2561 factory.go:103] Registering Raw factory Aug 05 17:32:53.916593 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.916466 2561 manager.go:1196] Started watching for new ooms in manager Aug 05 17:32:53.917100 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.917086 2561 manager.go:349] Starting recovery of all containers Aug 05 17:32:53.924616 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.924589 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 05 17:32:53.924778 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.924749 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 05 17:32:53.927602 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.927471 2561 manager.go:354] Recovery completed Aug 05 17:32:53.931846 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.931834 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:53.934281 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934253 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:53.934368 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934292 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:53.934368 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934303 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:53.934803 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934789 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 05 17:32:53.934803 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934802 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 05 17:32:53.934919 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934803 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:53.934919 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.934816 2561 state_mem.go:36] "Initialized new in-memory state store" Aug 05 17:32:53.935432 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.935419 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:53.935495 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.935441 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:53.935495 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.935450 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:53.936112 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.936101 2561 policy_none.go:49] "None policy: Start" Aug 05 17:32:53.936164 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.936117 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 05 17:32:53.936164 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.936126 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 05 17:32:53.937178 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.937168 2561 policy_none.go:47] "Start" Aug 05 17:32:53.945306 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.945183 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-87.ec2.internal.18c8f9215c8b6c28 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-87.ec2.internal,UID:ip-10-0-143-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-87.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-87.ec2.internal,},FirstTimestamp:2026-08-05 17:32:53.93427972 +0000 UTC m=+0.410562176,LastTimestamp:2026-08-05 17:32:53.93427972 +0000 UTC m=+0.410562176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-87.ec2.internal,}" Aug 05 17:32:53.946102 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.946087 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7xsq9" Aug 05 17:32:53.954423 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.954401 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7xsq9" Aug 05 17:32:53.956024 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.955943 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-87.ec2.internal.18c8f9215c8bb080 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-87.ec2.internal,UID:ip-10-0-143-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-87.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-87.ec2.internal,},FirstTimestamp:2026-08-05 17:32:53.934297216 +0000 UTC m=+0.410579671,LastTimestamp:2026-08-05 17:32:53.934297216 +0000 UTC m=+0.410579671,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-87.ec2.internal,}" Aug 05 17:32:53.968760 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.968747 2561 manager.go:337] "Starting Device Plugin manager" Aug 05 17:32:53.968838 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.968776 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 05 17:32:53.968838 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.968785 2561 server.go:87] "Starting device plugin registration server" Aug 05 17:32:53.969033 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.969012 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 05 17:32:53.969076 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.969026 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 05 17:32:53.969130 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.969111 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 05 17:32:53.969214 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.969199 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 05 17:32:53.969214 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:53.969213 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 05 17:32:53.969808 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.969781 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 05 17:32:53.969868 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:53.969834 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.061691 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.061654 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 05 17:32:54.062913 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.062892 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 05 17:32:54.063008 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.062933 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 05 17:32:54.063008 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.062957 2561 kubelet.go:2448] "Starting kubelet main sync loop" Aug 05 17:32:54.063008 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.062996 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 05 17:32:54.065903 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.065884 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:54.069881 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.069867 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:54.070645 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.070628 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:54.070727 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.070660 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:54.070727 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.070670 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:54.070727 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.070692 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.086729 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.086675 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.086729 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.086702 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-87.ec2.internal\": node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.131943 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.131913 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.163705 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.163675 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal"] Aug 05 17:32:54.163794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.163745 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:54.165273 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.165257 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:54.165327 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.165291 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:54.165327 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.165307 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:54.166520 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.166506 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:54.166657 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.166644 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.166696 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.166674 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:54.168127 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168112 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:54.168188 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168139 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:54.168188 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168152 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:54.168282 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168112 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:54.168282 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168214 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:54.168282 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.168245 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:54.169439 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.169425 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.169489 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.169455 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:54.170121 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.170101 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:54.170206 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.170133 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:54.170206 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.170143 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:54.191923 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.191892 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-87.ec2.internal\" not found" node="ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.196171 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.196153 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-87.ec2.internal\" not found" node="ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.218173 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.218144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.218272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.218179 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.218272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.218200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f01ba913ced227a9afb502c83707f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-87.ec2.internal\" (UID: \"6f01ba913ced227a9afb502c83707f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.232274 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.232254 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.318470 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f01ba913ced227a9afb502c83707f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-87.ec2.internal\" (UID: \"6f01ba913ced227a9afb502c83707f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.318612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.318612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318501 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.318612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318536 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.318612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318537 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f01ba913ced227a9afb502c83707f6b-config\") pod \"kube-apiserver-proxy-ip-10-0-143-87.ec2.internal\" (UID: \"6f01ba913ced227a9afb502c83707f6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.318612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.318546 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22737b7b18625200ca3673ffc43a2a8e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal\" (UID: \"22737b7b18625200ca3673ffc43a2a8e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.332564 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.332532 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.433325 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.433255 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.494493 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.494467 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.498835 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.498704 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:54.533798 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.533771 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.634370 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.634323 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.734857 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.734787 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.827946 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.827916 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:54.835125 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.835099 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.854313 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.854280 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 05 17:32:54.854473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.854452 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 17:32:54.854532 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.854460 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 05 17:32:54.935414 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:54.935377 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:54.958553 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.958517 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:27:54 +0000 UTC" deadline="2028-04-02 07:26:41.259372191 +0000 UTC" Aug 05 17:32:54.958553 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:54.958552 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14533h53m46.300824252s" Aug 05 17:32:55.013844 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.013814 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 05 17:32:55.015205 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.015181 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientMemory" Aug 05 17:32:55.015410 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.015246 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasNoDiskPressure" Aug 05 17:32:55.015410 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.015266 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeHasSufficientPID" Aug 05 17:32:55.035618 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.035583 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:55.049685 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:55.049641 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f01ba913ced227a9afb502c83707f6b.slice/crio-53ce25aa8c70a0f97164d2475c5ee4c11747ae9fcae66c80a95d106b4f5ee6c7 WatchSource:0}: Error finding container 53ce25aa8c70a0f97164d2475c5ee4c11747ae9fcae66c80a95d106b4f5ee6c7: Status 404 returned error can't find the container with id 53ce25aa8c70a0f97164d2475c5ee4c11747ae9fcae66c80a95d106b4f5ee6c7 Aug 05 17:32:55.050176 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:55.050158 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22737b7b18625200ca3673ffc43a2a8e.slice/crio-765efa5dfccc6938b168d9dc16a7bb1842ada9f99190083c15c86746c6f84517 WatchSource:0}: Error finding container 765efa5dfccc6938b168d9dc16a7bb1842ada9f99190083c15c86746c6f84517: Status 404 returned error can't find the container with id 765efa5dfccc6938b168d9dc16a7bb1842ada9f99190083c15c86746c6f84517 Aug 05 17:32:55.055963 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.055946 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:32:55.066219 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.066167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" event={"ID":"22737b7b18625200ca3673ffc43a2a8e","Type":"ContainerStarted","Data":"765efa5dfccc6938b168d9dc16a7bb1842ada9f99190083c15c86746c6f84517"} Aug 05 17:32:55.067197 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.067171 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" event={"ID":"6f01ba913ced227a9afb502c83707f6b","Type":"ContainerStarted","Data":"53ce25aa8c70a0f97164d2475c5ee4c11747ae9fcae66c80a95d106b4f5ee6c7"} Aug 05 17:32:55.136313 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.136271 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:55.216085 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.216055 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:55.237181 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.237145 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:55.337630 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.337539 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-87.ec2.internal\" not found" Aug 05 17:32:55.377212 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.377182 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:55.416039 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.415996 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" Aug 05 17:32:55.427793 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.427772 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:32:55.429068 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.429054 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" Aug 05 17:32:55.441204 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.441181 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 05 17:32:55.898175 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.898139 2561 apiserver.go:52] "Watching apiserver" Aug 05 17:32:55.907909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.907883 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 05 17:32:55.909702 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.909664 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-wcbrh","openshift-ovn-kubernetes/ovnkube-node-pv2lt","kube-system/global-pull-secret-syncer-4mfcd","kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z","openshift-cluster-node-tuning-operator/tuned-75t68","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal","openshift-network-diagnostics/network-check-target-kdr45","openshift-network-operator/iptables-alerter-wpkkz","kube-system/konnectivity-agent-6hk6k","openshift-dns/node-resolver-w965k","openshift-image-registry/node-ca-n5t2w","openshift-multus/multus-additional-cni-plugins-djr4b","openshift-multus/multus-f88s8"] Aug 05 17:32:55.911589 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.911539 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:55.914093 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.913642 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:55.915154 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.915129 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 05 17:32:55.915267 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.915195 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cp654\"" Aug 05 17:32:55.915358 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.915136 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 05 17:32:55.915672 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.915653 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.915901 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.916730 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.916951 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.917133 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.917546 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rfpd7\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.917942 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.918211 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-27cxz\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.918577 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.918881 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.919192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.919019 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.919771 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.919338 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.921158 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.919342 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 05 17:32:55.921158 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.920262 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:55.921158 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.920374 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:32:55.923817 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.921461 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:55.923817 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.921536 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:32:55.923817 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.922160 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rkks2\"" Aug 05 17:32:55.923817 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.922571 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 05 17:32:55.925433 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.924348 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.925639 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.925620 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:55.926814 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.926296 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 05 17:32:55.926814 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.926439 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 05 17:32:55.926814 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.926579 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 05 17:32:55.926814 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.926630 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.926814 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.926681 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.927868 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.927819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f88s8" Aug 05 17:32:55.928820 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.928740 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.928958 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929009 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929097 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929261 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929450 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2s8jt\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88736750-2f0f-4a39-a0db-075d627e9b1d-agent-certs\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929538 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-kubelet\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929619 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zb8qq\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-os-release\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929821 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-bin\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-registration-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929946 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d868l\" (UniqueName: \"kubernetes.io/projected/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kube-api-access-d868l\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.929977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-ovn\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930003 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-config\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930021 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcnzs\"" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930035 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovn-node-metrics-cert\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-script-lib\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930100 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.930409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-socket-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88736750-2f0f-4a39-a0db-075d627e9b1d-konnectivity-ca\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930202 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc46f\" (UniqueName: \"kubernetes.io/projected/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-kube-api-access-vc46f\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930267 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-device-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930279 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 05 17:32:55.931363 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.930339 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-cnibin\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.932028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.931995 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.932109 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932033 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.932109 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932109 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-netd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932107 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-host\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-serviceca\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-systemd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932181 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-var-lib-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932204 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-etc-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-sys-fs\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:55.932257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932252 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-systemd-units\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-slash\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmndm\" (UniqueName: \"kubernetes.io/projected/3892ff68-d082-4d8f-b0c6-7114e5137d60-kube-api-access-jmndm\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-binary-copy\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932373 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-dbus\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932402 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932431 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfrv\" (UniqueName: \"kubernetes.io/projected/a58be3a2-1580-4075-823e-05c7287b9b71-kube-api-access-xlfrv\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932684 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-netns\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.932710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-node-log\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932732 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-env-overrides\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932756 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-system-cni-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932780 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2l7r\" (UniqueName: \"kubernetes.io/projected/cc290291-e2cb-42d0-a874-0d96b118fe28-kube-api-access-t2l7r\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932869 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-kubelet-config\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-log-socket\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.933170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.932941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:55.934452 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.934428 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:55.934546 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:55.934517 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:32:55.934678 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.934658 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 05 17:32:55.935418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.935397 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:55.937347 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.937330 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:55.938536 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.938515 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 05 17:32:55.938737 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.938622 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.938835 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.938749 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mwnf7\"" Aug 05 17:32:55.938896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.938877 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.941717 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.941050 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 05 17:32:55.941717 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.941279 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 05 17:32:55.941717 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:55.941611 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wmz8g\"" Aug 05 17:32:56.012276 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.012227 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8t27" Aug 05 17:32:56.018360 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.018267 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 05 17:32:56.025276 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.025226 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z8t27" Aug 05 17:32:56.033748 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-kubelet\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.033905 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033757 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-daemon-config\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.033905 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033805 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-node-log\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.033905 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033834 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-env-overrides\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.033905 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033863 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2l7r\" (UniqueName: \"kubernetes.io/projected/cc290291-e2cb-42d0-a874-0d96b118fe28-kube-api-access-t2l7r\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.033905 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-kubelet-config\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033910 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-node-log\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033938 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cni-binary-copy\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.033989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034057 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034089 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-socket-dir-parent\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034116 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-netns\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-multus\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-bin\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034171 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-run\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-bin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovn-node-metrics-cert\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034239 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-socket-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfrv\" (UniqueName: \"kubernetes.io/projected/a58be3a2-1580-4075-823e-05c7287b9b71-kube-api-access-xlfrv\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-hosts-file\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-cnibin\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.034379 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034388 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034396 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-env-overrides\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.034454 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:32:56.534434603 +0000 UTC m=+3.010717064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:56.034485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-sys\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034504 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-binary-copy\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034533 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx7zf\" (UniqueName: \"kubernetes.io/projected/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-kube-api-access-rx7zf\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-modprobe-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-serviceca\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034615 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-var-lib-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034625 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-bin\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034640 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-etc-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-sys-fs\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-lib-modules\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.034721 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-slash\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.034766 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:56.534751698 +0000 UTC m=+3.011034148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034786 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-slash\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmndm\" (UniqueName: \"kubernetes.io/projected/3892ff68-d082-4d8f-b0c6-7114e5137d60-kube-api-access-jmndm\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034820 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-var-lib-kubelet\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-system-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.034923 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034856 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034874 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cnibin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-hostroot\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034963 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-conf-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-multus-certs\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysconfig\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035046 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv72n\" (UniqueName: \"kubernetes.io/projected/c2236c5f-99d0-475c-b1f0-018479f8f805-kube-api-access-fv72n\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035087 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-os-release\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-cnibin\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035092 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.034563 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-etc-selinux\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035155 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035211 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-kubelet-config\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035223 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-sys-fs\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-etc-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.035473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035282 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-var-lib-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035530 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-socket-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035692 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-netns\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035723 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-system-cni-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035766 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-kubernetes\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-tuned\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-k8s-cni-cncf-io\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-log-socket\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035894 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035901 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc290291-e2cb-42d0-a874-0d96b118fe28-cni-binary-copy\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88736750-2f0f-4a39-a0db-075d627e9b1d-agent-certs\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-kubelet\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035954 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-serviceca\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-log-socket\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035990 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-system-cni-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.035977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-os-release\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.036139 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc361f8-3a47-4879-b4c3-75bc236c11df-iptables-alerter-script\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036039 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-kubelet\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036057 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-systemd\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-os-release\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-etc-kubernetes\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-registration-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036151 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-netns\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036185 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d868l\" (UniqueName: \"kubernetes.io/projected/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kube-api-access-d868l\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036349 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-registration-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036374 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-conf\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-ovn\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036436 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-config\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-script-lib\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036498 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc361f8-3a47-4879-b4c3-75bc236c11df-host-slash\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036529 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vgj\" (UniqueName: \"kubernetes.io/projected/8cc361f8-3a47-4879-b4c3-75bc236c11df-kube-api-access-26vgj\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036556 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-tmp\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.037028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88736750-2f0f-4a39-a0db-075d627e9b1d-konnectivity-ca\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036614 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc46f\" (UniqueName: \"kubernetes.io/projected/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-kube-api-access-vc46f\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036614 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-ovn\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036692 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-device-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-dbus\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-netd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-host\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-systemd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036863 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hq2\" (UniqueName: \"kubernetes.io/projected/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-kube-api-access-g7hq2\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036891 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-systemd-units\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036917 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-tmp-dir\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036946 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036976 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.036982 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-config\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037003 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-host\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.037959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037200 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88736750-2f0f-4a39-a0db-075d627e9b1d-konnectivity-ca\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037272 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-host\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037359 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-openvswitch\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-systemd-units\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037402 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-run-systemd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037457 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-device-dir\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037507 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovnkube-script-lib\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-cni-netd\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037577 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc290291-e2cb-42d0-a874-0d96b118fe28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037604 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3892ff68-d082-4d8f-b0c6-7114e5137d60-host-run-ovn-kubernetes\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.038818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.037679 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18f36257-69b8-428e-b68b-4913f2dad3cf-dbus\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.039218 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.039196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3892ff68-d082-4d8f-b0c6-7114e5137d60-ovn-node-metrics-cert\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.039370 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.039341 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88736750-2f0f-4a39-a0db-075d627e9b1d-agent-certs\") pod \"konnectivity-agent-6hk6k\" (UID: \"88736750-2f0f-4a39-a0db-075d627e9b1d\") " pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:56.046845 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.046819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfrv\" (UniqueName: \"kubernetes.io/projected/a58be3a2-1580-4075-823e-05c7287b9b71-kube-api-access-xlfrv\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:56.046845 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.046829 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2l7r\" (UniqueName: \"kubernetes.io/projected/cc290291-e2cb-42d0-a874-0d96b118fe28-kube-api-access-t2l7r\") pod \"multus-additional-cni-plugins-djr4b\" (UID: \"cc290291-e2cb-42d0-a874-0d96b118fe28\") " pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.050483 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.050454 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d868l\" (UniqueName: \"kubernetes.io/projected/25dcdd55-c6e9-45b2-adb7-87c516f3e77d-kube-api-access-d868l\") pod \"aws-ebs-csi-driver-node-qpg8z\" (UID: \"25dcdd55-c6e9-45b2-adb7-87c516f3e77d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.054964 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.054941 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmndm\" (UniqueName: \"kubernetes.io/projected/3892ff68-d082-4d8f-b0c6-7114e5137d60-kube-api-access-jmndm\") pod \"ovnkube-node-pv2lt\" (UID: \"3892ff68-d082-4d8f-b0c6-7114e5137d60\") " pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.057684 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.057657 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc46f\" (UniqueName: \"kubernetes.io/projected/e2436a4d-8c40-44f6-84c2-3c7c8663a80c-kube-api-access-vc46f\") pod \"node-ca-n5t2w\" (UID: \"e2436a4d-8c40-44f6-84c2-3c7c8663a80c\") " pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.137909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.137868 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138095 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.137915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-socket-dir-parent\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138095 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.137949 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-netns\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138095 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-socket-dir-parent\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138194 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-multus\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-run\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138256 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-bin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138326 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-hosts-file\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.138499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-sys\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx7zf\" (UniqueName: \"kubernetes.io/projected/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-kube-api-access-rx7zf\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.138499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-modprobe-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138475 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-lib-modules\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138510 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-var-lib-kubelet\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-system-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cnibin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-hostroot\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-conf-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138681 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-multus-certs\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysconfig\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv72n\" (UniqueName: \"kubernetes.io/projected/c2236c5f-99d0-475c-b1f0-018479f8f805-kube-api-access-fv72n\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-os-release\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-kubernetes\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138797 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-tuned\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-k8s-cni-cncf-io\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.138885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138861 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc361f8-3a47-4879-b4c3-75bc236c11df-iptables-alerter-script\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138885 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-systemd\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138920 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-etc-kubernetes\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-conf\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.138988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc361f8-3a47-4879-b4c3-75bc236c11df-host-slash\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vgj\" (UniqueName: \"kubernetes.io/projected/8cc361f8-3a47-4879-b4c3-75bc236c11df-kube-api-access-26vgj\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139048 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-tmp\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hq2\" (UniqueName: \"kubernetes.io/projected/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-kube-api-access-g7hq2\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139116 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-tmp-dir\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-host\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139191 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-kubelet\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139247 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-daemon-config\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139290 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cni-binary-copy\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139444 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:56.139781 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139667 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-netns\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139781 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139701 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-multus\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139781 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139738 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-run\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.139781 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139762 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-cni-bin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.139991 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139791 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-hosts-file\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.139991 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.139821 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-sys\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.140098 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-modprobe-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.140184 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-lib-modules\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.140348 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140207 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-var-lib-kubelet\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.140348 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140245 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-system-cni-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.140348 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140273 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cnibin\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.140348 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140330 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-hostroot\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.140576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-conf-dir\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.140576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140419 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-multus-certs\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.140576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140486 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysconfig\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.140763 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140741 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8cc361f8-3a47-4879-b4c3-75bc236c11df-host-slash\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.140850 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.140835 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-os-release\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.141111 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141084 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-kubernetes\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141225 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-host\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141333 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-d\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141503 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-tmp-dir\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141821 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-var-lib-kubelet\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141898 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-systemd\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141904 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-cni-binary-copy\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141943 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-host-run-k8s-cni-cncf-io\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.141987 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-multus-daemon-config\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.142060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.142059 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-sysctl-conf\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.142621 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.142079 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-etc-kubernetes\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.142621 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.142496 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8cc361f8-3a47-4879-b4c3-75bc236c11df-iptables-alerter-script\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.143886 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.143853 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-etc-tuned\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.144938 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.144914 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2236c5f-99d0-475c-b1f0-018479f8f805-tmp\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.151253 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.151177 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:32:56.151253 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.151212 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:32:56.151253 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.151225 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:56.151454 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.151315 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:56.651296181 +0000 UTC m=+3.127578642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:56.157019 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.156991 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx7zf\" (UniqueName: \"kubernetes.io/projected/1382e3a7-4675-4b1c-a5a3-a193ab8f54a5-kube-api-access-rx7zf\") pod \"node-resolver-w965k\" (UID: \"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5\") " pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.162213 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.162181 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vgj\" (UniqueName: \"kubernetes.io/projected/8cc361f8-3a47-4879-b4c3-75bc236c11df-kube-api-access-26vgj\") pod \"iptables-alerter-wpkkz\" (UID: \"8cc361f8-3a47-4879-b4c3-75bc236c11df\") " pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.163634 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.163612 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hq2\" (UniqueName: \"kubernetes.io/projected/bf7ec92f-cb51-4e27-a80f-853a5ba8d719-kube-api-access-g7hq2\") pod \"multus-f88s8\" (UID: \"bf7ec92f-cb51-4e27-a80f-853a5ba8d719\") " pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.164728 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.164703 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv72n\" (UniqueName: \"kubernetes.io/projected/c2236c5f-99d0-475c-b1f0-018479f8f805-kube-api-access-fv72n\") pod \"tuned-75t68\" (UID: \"c2236c5f-99d0-475c-b1f0-018479f8f805\") " pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.230464 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.230424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:32:56.243889 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.243855 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5t2w" Aug 05 17:32:56.249385 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.249364 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:56.255280 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.255253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djr4b" Aug 05 17:32:56.258344 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.258318 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:32:56.266180 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.266153 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" Aug 05 17:32:56.274515 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.274488 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75t68" Aug 05 17:32:56.282446 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.282423 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f88s8" Aug 05 17:32:56.292233 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.292211 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpkkz" Aug 05 17:32:56.298150 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.298129 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w965k" Aug 05 17:32:56.542248 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.542196 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:56.542466 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.542283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:56.542466 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.542366 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:56.542466 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.542412 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:56.542466 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.542444 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:32:57.542423453 +0000 UTC m=+4.018705898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:56.542466 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.542470 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:57.542453885 +0000 UTC m=+4.018736329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:56.744159 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:56.744125 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:56.744344 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.744290 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:32:56.744344 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.744306 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:32:56.744344 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.744317 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:56.744498 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:56.744382 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:57.744369283 +0000 UTC m=+4.220651729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:57.028497 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.028450 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:27:56 +0000 UTC" deadline="2028-05-18 11:56:32.020935786 +0000 UTC" Aug 05 17:32:57.028497 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.028484 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15642h23m34.992455375s" Aug 05 17:32:57.069472 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:57.069440 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2236c5f_99d0_475c_b1f0_018479f8f805.slice/crio-33cdd68a406a0c72c868d6be3230ff510b8af84f60477832277158be7fd3dcb9 WatchSource:0}: Error finding container 33cdd68a406a0c72c868d6be3230ff510b8af84f60477832277158be7fd3dcb9: Status 404 returned error can't find the container with id 33cdd68a406a0c72c868d6be3230ff510b8af84f60477832277158be7fd3dcb9 Aug 05 17:32:57.070951 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:57.070919 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1382e3a7_4675_4b1c_a5a3_a193ab8f54a5.slice/crio-ec92a57f1bd4cd1dc813df0f4c20d6dafcaa9241c27f3a683b7bd58ebb8230bd WatchSource:0}: Error finding container ec92a57f1bd4cd1dc813df0f4c20d6dafcaa9241c27f3a683b7bd58ebb8230bd: Status 404 returned error can't find the container with id ec92a57f1bd4cd1dc813df0f4c20d6dafcaa9241c27f3a683b7bd58ebb8230bd Aug 05 17:32:57.072157 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:57.072135 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc290291_e2cb_42d0_a874_0d96b118fe28.slice/crio-980898fb203f611fa8f9747598db27be467fa62ff04c3a6e7fc15c5e56092010 WatchSource:0}: Error finding container 980898fb203f611fa8f9747598db27be467fa62ff04c3a6e7fc15c5e56092010: Status 404 returned error can't find the container with id 980898fb203f611fa8f9747598db27be467fa62ff04c3a6e7fc15c5e56092010 Aug 05 17:32:57.073337 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.073172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75t68" event={"ID":"c2236c5f-99d0-475c-b1f0-018479f8f805","Type":"ContainerStarted","Data":"33cdd68a406a0c72c868d6be3230ff510b8af84f60477832277158be7fd3dcb9"} Aug 05 17:32:57.075495 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:57.075470 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25dcdd55_c6e9_45b2_adb7_87c516f3e77d.slice/crio-d4361a0531a1b1567665e8739601dc107061cc56f6ede3fffa7cbf087c6e794b WatchSource:0}: Error finding container d4361a0531a1b1567665e8739601dc107061cc56f6ede3fffa7cbf087c6e794b: Status 404 returned error can't find the container with id d4361a0531a1b1567665e8739601dc107061cc56f6ede3fffa7cbf087c6e794b Aug 05 17:32:57.077108 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:32:57.077075 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88736750_2f0f_4a39_a0db_075d627e9b1d.slice/crio-7688ed3b72db2c3e4f28fea89fb27e739f1c71ecd628630e9d86956467bfa2b0 WatchSource:0}: Error finding container 7688ed3b72db2c3e4f28fea89fb27e739f1c71ecd628630e9d86956467bfa2b0: Status 404 returned error can't find the container with id 7688ed3b72db2c3e4f28fea89fb27e739f1c71ecd628630e9d86956467bfa2b0 Aug 05 17:32:57.549351 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.549233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:57.549528 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.549354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:57.549528 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.549418 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:57.549528 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.549455 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:57.549528 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.549495 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:59.549470621 +0000 UTC m=+6.025753068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:57.549528 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.549512 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:32:59.549505966 +0000 UTC m=+6.025788410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:57.751663 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:57.751383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:57.751663 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.751586 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:32:57.751663 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.751614 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:32:57.751663 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.751629 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:57.752027 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:57.751693 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:59.751674438 +0000 UTC m=+6.227956899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:58.013247 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.013210 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 05 17:32:58.029357 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.029265 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:27:56 +0000 UTC" deadline="2028-04-23 08:30:52.194560106 +0000 UTC" Aug 05 17:32:58.029357 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.029343 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15038h57m54.165221555s" Aug 05 17:32:58.065624 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.065434 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:58.065624 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.065479 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:58.065624 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:58.065580 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:32:58.065916 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:58.065654 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:32:58.065916 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.065720 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:58.065916 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:58.065796 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:32:58.098031 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.097533 2561 generic.go:356] "Generic (PLEG): container finished" podID="22737b7b18625200ca3673ffc43a2a8e" containerID="5a08d3c781ae6d068c453e485d0af11b425dbe866206db9db6f3bbdec64f1c05" exitCode=0 Aug 05 17:32:58.098031 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.097627 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" event={"ID":"22737b7b18625200ca3673ffc43a2a8e","Type":"ContainerDied","Data":"5a08d3c781ae6d068c453e485d0af11b425dbe866206db9db6f3bbdec64f1c05"} Aug 05 17:32:58.113440 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.113404 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" event={"ID":"6f01ba913ced227a9afb502c83707f6b","Type":"ContainerStarted","Data":"c353fac7c53a03fc483faea0b41dc69e4d76e626531ff074fb0e17fe2c8ca25d"} Aug 05 17:32:58.136762 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.136730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" event={"ID":"25dcdd55-c6e9-45b2-adb7-87c516f3e77d","Type":"ContainerStarted","Data":"d4361a0531a1b1567665e8739601dc107061cc56f6ede3fffa7cbf087c6e794b"} Aug 05 17:32:58.155013 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.154976 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"e4ef9d9e5ab1b001cda5d8361a1e984c52d54d1737e559751061817d5a7a37e0"} Aug 05 17:32:58.171935 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.171796 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6hk6k" event={"ID":"88736750-2f0f-4a39-a0db-075d627e9b1d","Type":"ContainerStarted","Data":"7688ed3b72db2c3e4f28fea89fb27e739f1c71ecd628630e9d86956467bfa2b0"} Aug 05 17:32:58.176975 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.176868 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpkkz" event={"ID":"8cc361f8-3a47-4879-b4c3-75bc236c11df","Type":"ContainerStarted","Data":"71b01d6afaf234210ee01b2a0d2c0cb1f8a47df01ce00321ae6b4934cb2625c7"} Aug 05 17:32:58.183324 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.183208 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f88s8" event={"ID":"bf7ec92f-cb51-4e27-a80f-853a5ba8d719","Type":"ContainerStarted","Data":"1af5dd638cca925b9a90e0f053190ad639118dada5a388a47925e78855317e4d"} Aug 05 17:32:58.200012 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.199984 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5t2w" event={"ID":"e2436a4d-8c40-44f6-84c2-3c7c8663a80c","Type":"ContainerStarted","Data":"9f259ed4847c81d97c3c40e83f1a614cfc92d5a6398a01135d41c91bf7435e97"} Aug 05 17:32:58.218298 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.218250 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerStarted","Data":"980898fb203f611fa8f9747598db27be467fa62ff04c3a6e7fc15c5e56092010"} Aug 05 17:32:58.224937 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:58.224896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w965k" event={"ID":"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5","Type":"ContainerStarted","Data":"ec92a57f1bd4cd1dc813df0f4c20d6dafcaa9241c27f3a683b7bd58ebb8230bd"} Aug 05 17:32:59.242829 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:59.242387 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" event={"ID":"22737b7b18625200ca3673ffc43a2a8e","Type":"ContainerStarted","Data":"b588557dfa0a95812e9fc0c8cbe3d310f8412a357a666e59ac419ce07fa9922d"} Aug 05 17:32:59.271496 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:59.271432 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-87.ec2.internal" podStartSLOduration=3.271408751 podStartE2EDuration="3.271408751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:32:58.151337555 +0000 UTC m=+4.627620025" watchObservedRunningTime="2026-08-05 17:32:59.271408751 +0000 UTC m=+5.747691203" Aug 05 17:32:59.569100 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:59.569058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:32:59.569330 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:59.569127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:32:59.569330 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.569276 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:59.569460 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.569349 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:03.569325845 +0000 UTC m=+10.045608310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:32:59.569836 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.569813 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:59.569946 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.569874 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:33:03.569857878 +0000 UTC m=+10.046140327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:32:59.771424 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:32:59.771375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:32:59.771595 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.771580 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:32:59.771663 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.771604 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:32:59.771663 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.771618 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:32:59.771794 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:32:59.771679 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:03.77165937 +0000 UTC m=+10.247941839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:00.064435 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:00.064400 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:00.064435 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:00.064418 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:00.064716 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:00.064550 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:00.064716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:00.064557 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:00.064716 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:00.064686 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:00.064923 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:00.064795 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:02.065205 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:02.065154 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:02.065773 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:02.065278 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:02.065773 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:02.065368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:02.065773 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:02.065431 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:02.065773 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:02.065473 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:02.065773 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:02.065542 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:03.602059 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:03.602020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:03.602564 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:03.602083 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:03.602564 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.602180 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:03.602564 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.602196 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:03.602564 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.602258 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:11.60224066 +0000 UTC m=+18.078523105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:03.602564 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.602297 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:33:11.602286525 +0000 UTC m=+18.078568970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:03.803692 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:03.803641 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:03.803870 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.803830 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:33:03.803870 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.803848 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:33:03.803870 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.803859 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:03.804064 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:03.803917 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:11.80389581 +0000 UTC m=+18.280178255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:04.064497 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:04.064415 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:04.064497 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:04.064429 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:04.065869 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:04.065182 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:04.065869 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:04.065547 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:04.065869 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:04.065603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:04.065869 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:04.065662 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:06.067933 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:06.067895 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:06.068402 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:06.068015 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:06.068402 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:06.068070 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:06.068402 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:06.068137 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:06.068402 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:06.068185 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:06.068402 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:06.068274 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:08.064108 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:08.064075 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:08.064582 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:08.064076 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:08.064582 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:08.064200 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:08.064582 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:08.064286 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:08.064582 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:08.064076 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:08.064582 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:08.064439 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:10.063944 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:10.063904 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:10.064379 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:10.063904 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:10.064379 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:10.064048 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:10.064379 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:10.063922 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:10.064379 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:10.064128 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:10.064379 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:10.064204 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:11.666944 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:11.666900 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:11.667483 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:11.666972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:11.667483 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.667059 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:11.667483 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.667090 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:11.667483 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.667129 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:33:27.667113965 +0000 UTC m=+34.143396409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:11.667483 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.667159 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:27.667139714 +0000 UTC m=+34.143422163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:11.868831 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:11.868785 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:11.869081 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.868986 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:33:11.869081 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.869018 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:33:11.869081 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.869036 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:11.869265 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:11.869106 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:27.869085916 +0000 UTC m=+34.345368363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:12.067201 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:12.067164 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:12.067426 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:12.067208 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:12.067426 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:12.067304 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:12.067426 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:12.067320 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:12.067426 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:12.067379 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:12.067619 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:12.067469 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:14.064046 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:14.064004 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:14.064579 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:14.064116 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:14.064579 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:14.064118 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:14.064579 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:14.064188 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:14.064579 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:14.064259 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:14.064579 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:14.064352 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:16.063468 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:16.063433 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:16.063934 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:16.063432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:16.063934 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:16.063556 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:16.063934 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:16.063432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:16.063934 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:16.063624 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:16.063934 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:16.063656 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:17.287152 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.286726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5t2w" event={"ID":"e2436a4d-8c40-44f6-84c2-3c7c8663a80c","Type":"ContainerStarted","Data":"3c07a752c7108fafd17f056d188f89b4b485a3b2e509739dd50b69d80c8d21de"} Aug 05 17:33:17.288455 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.288423 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="edc724b78d15881d751477b867014c8f6acc09a6d351ac3fda1a5d7faba3007b" exitCode=0 Aug 05 17:33:17.288570 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.288500 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"edc724b78d15881d751477b867014c8f6acc09a6d351ac3fda1a5d7faba3007b"} Aug 05 17:33:17.290133 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.290001 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w965k" event={"ID":"1382e3a7-4675-4b1c-a5a3-a193ab8f54a5","Type":"ContainerStarted","Data":"05dc934c45a50ee9c9ff55a98bf87dedc03f9c5c9f15837e723aab23a9e1648f"} Aug 05 17:33:17.291568 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.291515 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" event={"ID":"25dcdd55-c6e9-45b2-adb7-87c516f3e77d","Type":"ContainerStarted","Data":"254d78d3614d83ba42f26dfa84df14083c31d2cd0445d3e608eafdbf19bc1f44"} Aug 05 17:33:17.293028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.293000 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75t68" event={"ID":"c2236c5f-99d0-475c-b1f0-018479f8f805","Type":"ContainerStarted","Data":"8887863080e35d50bb84362015bd933ee6ab5b573b09023776b42da952c15fb3"} Aug 05 17:33:17.295021 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.294999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"47c16742f1a0c351370131107b5331e2b5467062e29ce4a1122e54c73f8186ab"} Aug 05 17:33:17.295106 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.295050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"b745b726a034f637a329fe7dcf3811eebfce4bc8a56842ff2f0fbe1f894e4293"} Aug 05 17:33:17.295106 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.295083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"d4eaa0fa547e909888a670481339b119980c2b5fbf9ce2922cd1692bbeae26a5"} Aug 05 17:33:17.296540 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.296519 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6hk6k" event={"ID":"88736750-2f0f-4a39-a0db-075d627e9b1d","Type":"ContainerStarted","Data":"c5dd92e66713fa61f54e7258c8eaacd9ee4bde6d3aa77b921f454629e963eb1d"} Aug 05 17:33:17.298355 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.298326 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f88s8" event={"ID":"bf7ec92f-cb51-4e27-a80f-853a5ba8d719","Type":"ContainerStarted","Data":"7582b90422fe8b5754c9a55a1b535abc44ef4ff15963f57e39794888e3913538"} Aug 05 17:33:17.306844 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.306806 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-87.ec2.internal" podStartSLOduration=21.306793136 podStartE2EDuration="21.306793136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:32:59.27278811 +0000 UTC m=+5.749070580" watchObservedRunningTime="2026-08-05 17:33:17.306793136 +0000 UTC m=+23.783075617" Aug 05 17:33:17.307010 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.306993 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n5t2w" podStartSLOduration=3.475301176 podStartE2EDuration="23.306989034s" totalImagesPullingTime="19.831687858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:17.306139648 +0000 UTC m=+23.782422112" watchObservedRunningTime="2026-08-05 17:33:17.306989034 +0000 UTC m=+23.783271498" Aug 05 17:33:17.329218 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.328883 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f88s8" podStartSLOduration=2.451383334 podStartE2EDuration="22.32884979s" totalImagesPullingTime="19.877466456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:17.328409445 +0000 UTC m=+23.804691911" watchObservedRunningTime="2026-08-05 17:33:17.32884979 +0000 UTC m=+23.805132255" Aug 05 17:33:17.352590 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.352537 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w965k" podStartSLOduration=2.515948227 podStartE2EDuration="22.352521546s" totalImagesPullingTime="19.836573319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:17.352110128 +0000 UTC m=+23.828392592" watchObservedRunningTime="2026-08-05 17:33:17.352521546 +0000 UTC m=+23.828804010" Aug 05 17:33:17.369400 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.369334 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6hk6k" podStartSLOduration=3.543795291 podStartE2EDuration="23.369317364s" totalImagesPullingTime="19.825522073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:17.368831803 +0000 UTC m=+23.845114272" watchObservedRunningTime="2026-08-05 17:33:17.369317364 +0000 UTC m=+23.845599832" Aug 05 17:33:17.414730 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:17.414681 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-75t68" podStartSLOduration=2.57396309 podStartE2EDuration="22.414666311s" totalImagesPullingTime="19.840703221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:17.414329389 +0000 UTC m=+23.890611865" watchObservedRunningTime="2026-08-05 17:33:17.414666311 +0000 UTC m=+23.890948775" Aug 05 17:33:18.063583 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.063549 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:18.063814 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:18.063746 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:18.063865 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.063551 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:18.063951 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:18.063936 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:18.064001 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.063549 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:18.064122 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:18.064105 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:18.304427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.304348 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"926a534dc5f9abdcc07dbae9cd0b2da01a18c1422e80694de8646342c6d05425"} Aug 05 17:33:18.304896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.304513 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"ec2675c8cb921e481accb8f095c7f692fa6f90040f15ce179ee3c301f8e3ed60"} Aug 05 17:33:18.304896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.304572 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"6fd37054d9f8e08860682c9f71eab53e7ed7b9b322f87a4c249ed8fa88a76495"} Aug 05 17:33:18.927370 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.927336 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 05 17:33:18.982437 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.982328 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:33:18.927363243Z","UUID":"2f573a5e-4bbc-42ed-86a2-e768cb69accc","Handler":null,"Name":"","Endpoint":""} Aug 05 17:33:18.984077 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.984054 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 05 17:33:18.984213 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:18.984087 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 05 17:33:19.307998 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:19.307931 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpkkz" event={"ID":"8cc361f8-3a47-4879-b4c3-75bc236c11df","Type":"ContainerStarted","Data":"9507a9dea08083146106906b3cc020fdfe0826e2f5c3a39a00fa1130f4187287"} Aug 05 17:33:19.309986 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:19.309959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" event={"ID":"25dcdd55-c6e9-45b2-adb7-87c516f3e77d","Type":"ContainerStarted","Data":"f0d113e60f8863e3968352c6a95d4c7a6cc377fa47eff5a6c97134f5c9f3c50d"} Aug 05 17:33:19.324782 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:19.324728 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wpkkz" podStartSLOduration=4.492960757 podStartE2EDuration="24.324713339s" totalImagesPullingTime="19.831752582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:19.324051705 +0000 UTC m=+25.800334171" watchObservedRunningTime="2026-08-05 17:33:19.324713339 +0000 UTC m=+25.800995803" Aug 05 17:33:20.063767 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:20.063736 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:20.064025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:20.063740 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:20.064025 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:20.063831 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:20.064025 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:20.063924 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:20.064025 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:20.063822 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:20.064185 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:20.064031 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:20.316396 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:20.316305 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"2c114c5bd8839b53663be56926ae6e9adb97dee513012a7439213e432e252138"} Aug 05 17:33:21.150180 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:21.149979 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:33:21.150724 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:21.150697 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:33:21.319527 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:21.319490 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:33:21.320210 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:21.320191 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6hk6k" Aug 05 17:33:22.063569 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.063534 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:22.063695 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.063534 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:22.063695 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:22.063677 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:22.063796 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:22.063753 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:22.063796 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.063533 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:22.063916 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:22.063842 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:22.323582 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.323319 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" event={"ID":"25dcdd55-c6e9-45b2-adb7-87c516f3e77d","Type":"ContainerStarted","Data":"c0878fcd193f3ff15a7db8b7f88ef230e01b4e1754f8df46e0febf4fcce062ea"} Aug 05 17:33:22.327088 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.327051 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" event={"ID":"3892ff68-d082-4d8f-b0c6-7114e5137d60","Type":"ContainerStarted","Data":"5f5a53dbc240f8bca5b261172ebb36e31f6c38ed9f72df4e6962f1ad86345833"} Aug 05 17:33:22.327300 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.327218 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:22.327369 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.327300 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:22.328920 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.328894 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="775cbaf340114994fe1c312a8c2f1196d6306177192370c35d48e62dd3ca7d6f" exitCode=0 Aug 05 17:33:22.329037 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.328943 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"775cbaf340114994fe1c312a8c2f1196d6306177192370c35d48e62dd3ca7d6f"} Aug 05 17:33:22.341418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.341380 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qpg8z" podStartSLOduration=3.04272072 podStartE2EDuration="27.341364538s" totalImagesPullingTime="24.298643818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:22.340872996 +0000 UTC m=+28.817155460" watchObservedRunningTime="2026-08-05 17:33:22.341364538 +0000 UTC m=+28.817647002" Aug 05 17:33:22.344028 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.344006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:22.369819 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:22.369774 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" podStartSLOduration=7.484825936 podStartE2EDuration="27.369759222s" totalImagesPullingTime="19.884933286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:22.369492311 +0000 UTC m=+28.845774776" watchObservedRunningTime="2026-08-05 17:33:22.369759222 +0000 UTC m=+28.846041687" Aug 05 17:33:23.331953 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:23.331925 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:23.348220 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:23.348156 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:24.064764 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.064721 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:24.064987 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.064816 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:24.064987 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.064860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:24.064987 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.064851 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:24.064987 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.064959 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:24.065122 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.065049 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:24.132959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.132920 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdr45"] Aug 05 17:33:24.135984 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.135929 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4mfcd"] Aug 05 17:33:24.136547 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.136513 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcbrh"] Aug 05 17:33:24.336274 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.336184 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="2a10a7c37367ba174b03e20d5d896c16a5828c30973319fa82f97b56313f3ea9" exitCode=0 Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.336271 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"2a10a7c37367ba174b03e20d5d896c16a5828c30973319fa82f97b56313f3ea9"} Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.336299 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.336288 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:24.336374 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.336505 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:24.336679 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.336562 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:24.337064 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:24.337010 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:26.067636 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:26.067425 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:26.068215 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:26.067423 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:26.068215 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:26.067748 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:26.068215 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:26.067448 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:26.068215 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:26.067788 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:26.068215 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:26.067869 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:26.343576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:26.343491 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="cfda452f00a79ee0d5f1c1128d3425b7b4c66f177b50f4e64cb71470374b9d84" exitCode=0 Aug 05 17:33:26.343576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:26.343551 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"cfda452f00a79ee0d5f1c1128d3425b7b4c66f177b50f4e64cb71470374b9d84"} Aug 05 17:33:27.689823 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:27.689790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:27.692846 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:27.689847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:27.692846 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.689942 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:27.692846 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.689949 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:27.692846 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.690013 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs podName:a58be3a2-1580-4075-823e-05c7287b9b71 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:59.689997295 +0000 UTC m=+66.166279739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs") pod "network-metrics-daemon-wcbrh" (UID: "a58be3a2-1580-4075-823e-05c7287b9b71") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 05 17:33:27.692846 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.690029 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret podName:18f36257-69b8-428e-b68b-4913f2dad3cf nodeName:}" failed. No retries permitted until 2026-08-05 17:33:59.690022725 +0000 UTC m=+66.166305167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret") pod "global-pull-secret-syncer-4mfcd" (UID: "18f36257-69b8-428e-b68b-4913f2dad3cf") : object "kube-system"/"original-pull-secret" not registered Aug 05 17:33:27.890866 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:27.890823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:27.891045 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.890982 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 05 17:33:27.891045 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.891008 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 05 17:33:27.891045 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.891018 2561 projected.go:196] Error preparing data for projected volume kube-api-access-tdkfz for pod openshift-network-diagnostics/network-check-target-kdr45: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:27.891185 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:27.891088 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz podName:ce729a7f-efb2-47bb-a2e2-d0877520c2b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:59.891067409 +0000 UTC m=+66.367349854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tdkfz" (UniqueName: "kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz") pod "network-check-target-kdr45" (UID: "ce729a7f-efb2-47bb-a2e2-d0877520c2b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 05 17:33:28.067148 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:28.067114 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:28.067148 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:28.067148 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:28.067415 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:28.067290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:28.067415 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:28.067285 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcbrh" podUID="a58be3a2-1580-4075-823e-05c7287b9b71" Aug 05 17:33:28.067415 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:28.067397 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kdr45" podUID="ce729a7f-efb2-47bb-a2e2-d0877520c2b8" Aug 05 17:33:28.067547 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:28.067473 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4mfcd" podUID="18f36257-69b8-428e-b68b-4913f2dad3cf" Aug 05 17:33:29.826786 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.826758 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-87.ec2.internal" event="NodeReady" Aug 05 17:33:29.827374 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.826887 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 05 17:33:29.861909 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.861874 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:33:29.864390 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.864372 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:29.866826 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.866584 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 05 17:33:29.866826 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.866666 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 05 17:33:29.866826 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.866730 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 05 17:33:29.867209 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.867188 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jvd4r\"" Aug 05 17:33:29.872850 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.872830 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 05 17:33:29.876940 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.876923 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nxfjd"] Aug 05 17:33:29.878852 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.878833 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:29.880190 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.880170 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:33:29.880304 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.880210 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nntms"] Aug 05 17:33:29.881404 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.881382 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 05 17:33:29.881607 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.881590 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59qnw\"" Aug 05 17:33:29.881710 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.881590 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 05 17:33:29.881891 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.881874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:29.884543 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.884514 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 05 17:33:29.884662 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.884638 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 05 17:33:29.884753 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.884719 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xvqv4\"" Aug 05 17:33:29.884791 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.884774 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 05 17:33:29.892245 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.892206 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxfjd"] Aug 05 17:33:29.892900 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:29.892879 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nntms"] Aug 05 17:33:30.005809 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.005768 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de59047-10b9-441d-814d-2654e9f013a9-tmp-dir\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.005993 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.005818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.005993 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.005952 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817f6abd-e663-4560-b4d8-a7eaf8697446-cert\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.006115 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006003 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2twk\" (UniqueName: \"kubernetes.io/projected/817f6abd-e663-4560-b4d8-a7eaf8697446-kube-api-access-g2twk\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.006115 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006028 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de59047-10b9-441d-814d-2654e9f013a9-config-volume\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.006115 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006048 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rls\" (UniqueName: \"kubernetes.io/projected/7de59047-10b9-441d-814d-2654e9f013a9-kube-api-access-m8rls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.006115 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006170 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9pv\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006270 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.006451 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.006361 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.063202 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.063155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:30.066418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.066395 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:30.066590 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.066554 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:30.066863 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.066656 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:30.069376 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.069137 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:33:30.069376 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.069310 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 05 17:33:30.070467 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.069893 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 05 17:33:30.070467 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.070214 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dt589\"" Aug 05 17:33:30.070467 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.070416 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bbv86\"" Aug 05 17:33:30.070628 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.070484 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:33:30.106778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.106731 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.106961 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.106789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.106961 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.106824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.106961 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.106916 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:30.107117 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.106977 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:30.606960835 +0000 UTC m=+37.083243279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:30.107117 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.106996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107117 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107117 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.107104 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:30.107117 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.107113 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.107141 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:30.607132059 +0000 UTC m=+37.083414501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de59047-10b9-441d-814d-2654e9f013a9-tmp-dir\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107313 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817f6abd-e663-4560-b4d8-a7eaf8697446-cert\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2twk\" (UniqueName: \"kubernetes.io/projected/817f6abd-e663-4560-b4d8-a7eaf8697446-kube-api-access-g2twk\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de59047-10b9-441d-814d-2654e9f013a9-config-volume\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.107387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rls\" (UniqueName: \"kubernetes.io/projected/7de59047-10b9-441d-814d-2654e9f013a9-kube-api-access-m8rls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.107716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107440 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9pv\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107481 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.107716 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107527 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de59047-10b9-441d-814d-2654e9f013a9-tmp-dir\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.107948 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.107805 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.108284 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.108222 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7de59047-10b9-441d-814d-2654e9f013a9-config-volume\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.108415 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.108397 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.112340 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.112314 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.112464 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.112323 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/817f6abd-e663-4560-b4d8-a7eaf8697446-cert\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.112623 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.112598 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.116562 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.116535 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.116671 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.116626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9pv\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.116841 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.116813 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rls\" (UniqueName: \"kubernetes.io/projected/7de59047-10b9-441d-814d-2654e9f013a9-kube-api-access-m8rls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.117152 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.117130 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2twk\" (UniqueName: \"kubernetes.io/projected/817f6abd-e663-4560-b4d8-a7eaf8697446-kube-api-access-g2twk\") pod \"ingress-canary-nntms\" (UID: \"817f6abd-e663-4560-b4d8-a7eaf8697446\") " pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.164167 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.164128 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:30.197092 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.197056 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nntms" Aug 05 17:33:30.264955 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.264914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:30.612174 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.612024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:30.612174 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.612173 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:30.612418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.612204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:30.612418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.612245 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:31.612213331 +0000 UTC m=+38.088495774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:30.612418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.612308 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:30.612418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.612322 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:30.612418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:30.612375 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:31.61235888 +0000 UTC m=+38.088641325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:30.643585 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:30.643561 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nntms"] Aug 05 17:33:30.646895 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:30.646870 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817f6abd_e663_4560_b4d8_a7eaf8697446.slice/crio-c04fe98f0f663d01becf2fc61ea10439a079389b4da1d84da7df02cb10f25d74 WatchSource:0}: Error finding container c04fe98f0f663d01becf2fc61ea10439a079389b4da1d84da7df02cb10f25d74: Status 404 returned error can't find the container with id c04fe98f0f663d01becf2fc61ea10439a079389b4da1d84da7df02cb10f25d74 Aug 05 17:33:31.360094 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:31.360056 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="5533f3548a1212a84581faf4700c9000ea6b79cf81901cb419beaa64ed646909" exitCode=0 Aug 05 17:33:31.360675 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:31.360135 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"5533f3548a1212a84581faf4700c9000ea6b79cf81901cb419beaa64ed646909"} Aug 05 17:33:31.361312 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:31.361284 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nntms" event={"ID":"817f6abd-e663-4560-b4d8-a7eaf8697446","Type":"ContainerStarted","Data":"c04fe98f0f663d01becf2fc61ea10439a079389b4da1d84da7df02cb10f25d74"} Aug 05 17:33:31.619707 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:31.619615 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:31.619707 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:31.619694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:31.619935 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:31.619756 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:31.619935 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:31.619819 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:33.619800617 +0000 UTC m=+40.096083061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:31.619935 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:31.619836 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:31.619935 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:31.619851 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:31.619935 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:31.619902 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:33.619886141 +0000 UTC m=+40.096168587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:32.369298 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.369258 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc290291-e2cb-42d0-a874-0d96b118fe28" containerID="67d7c963fc73b983d05e475b32a5d68e9f9933eb6036954acced4ffe10928d59" exitCode=0 Aug 05 17:33:32.369809 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.369307 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerDied","Data":"67d7c963fc73b983d05e475b32a5d68e9f9933eb6036954acced4ffe10928d59"} Aug 05 17:33:32.716949 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.716868 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-nrs7p"] Aug 05 17:33:32.738397 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.738370 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-nrs7p"] Aug 05 17:33:32.738546 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.738473 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.744910 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.744882 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 05 17:33:32.745018 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.744944 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 05 17:33:32.745109 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.745095 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:33:32.747116 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.747095 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 05 17:33:32.748379 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.748188 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ktkwr\"" Aug 05 17:33:32.753190 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.753172 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 05 17:33:32.830178 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.830145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-config\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.830301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.830183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-trusted-ca\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.830301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.830205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1be440ce-b9fe-486b-8edd-5fc930de9c3c-serving-cert\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.830397 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.830315 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7628\" (UniqueName: \"kubernetes.io/projected/1be440ce-b9fe-486b-8edd-5fc930de9c3c-kube-api-access-k7628\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.910328 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.910167 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82"] Aug 05 17:33:32.919247 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.919211 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86984d778b-h5jmk"] Aug 05 17:33:32.919389 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.919326 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" Aug 05 17:33:32.923377 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.923337 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f6972\"" Aug 05 17:33:32.931465 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.931436 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-config\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.931565 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.931483 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-trusted-ca\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.931565 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.931512 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1be440ce-b9fe-486b-8edd-5fc930de9c3c-serving-cert\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.931706 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.931684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7628\" (UniqueName: \"kubernetes.io/projected/1be440ce-b9fe-486b-8edd-5fc930de9c3c-kube-api-access-k7628\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.932214 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.932186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-config\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.932344 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.932201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82"] Aug 05 17:33:32.932344 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.932290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:32.932818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.932683 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1be440ce-b9fe-486b-8edd-5fc930de9c3c-trusted-ca\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.936572 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.936533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1be440ce-b9fe-486b-8edd-5fc930de9c3c-serving-cert\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:32.936890 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.936873 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 05 17:33:32.936982 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.936958 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 05 17:33:32.937046 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937000 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fvnv\"" Aug 05 17:33:32.937127 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937106 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 05 17:33:32.937374 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937152 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 05 17:33:32.937374 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937194 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 05 17:33:32.937374 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937259 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 05 17:33:32.937896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.937877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86984d778b-h5jmk"] Aug 05 17:33:32.954369 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:32.954346 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7628\" (UniqueName: \"kubernetes.io/projected/1be440ce-b9fe-486b-8edd-5fc930de9c3c-kube-api-access-k7628\") pod \"console-operator-5ccd6b445f-nrs7p\" (UID: \"1be440ce-b9fe-486b-8edd-5fc930de9c3c\") " pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:33.032172 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-default-certificate\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.032266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-stats-auth\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.032266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032198 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.032266 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnq2\" (UniqueName: \"kubernetes.io/projected/70b69e2d-e493-4ba2-9fb0-58a54d618ffc-kube-api-access-8dnq2\") pod \"network-check-source-65bdbbff45-sxq82\" (UID: \"70b69e2d-e493-4ba2-9fb0-58a54d618ffc\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" Aug 05 17:33:33.032403 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.032403 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.032395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2dt\" (UniqueName: \"kubernetes.io/projected/f7963633-3133-4b6d-af46-d0dd588a423c-kube-api-access-dm2dt\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.052217 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.052198 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:33.133287 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133256 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2dt\" (UniqueName: \"kubernetes.io/projected/f7963633-3133-4b6d-af46-d0dd588a423c-kube-api-access-dm2dt\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.133405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133389 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-default-certificate\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.133456 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-stats-auth\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.133456 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.133456 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnq2\" (UniqueName: \"kubernetes.io/projected/70b69e2d-e493-4ba2-9fb0-58a54d618ffc-kube-api-access-8dnq2\") pod \"network-check-source-65bdbbff45-sxq82\" (UID: \"70b69e2d-e493-4ba2-9fb0-58a54d618ffc\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" Aug 05 17:33:33.133758 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.133671 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:33:33.133826 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.133815 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:33.633793113 +0000 UTC m=+40.110075574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : secret "router-metrics-certs-default" not found Aug 05 17:33:33.133891 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.133843 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:33.633823668 +0000 UTC m=+40.110106114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:33.133891 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.133745 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.136418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.136395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-stats-auth\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.136418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.136411 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-default-certificate\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.144647 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.144611 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnq2\" (UniqueName: \"kubernetes.io/projected/70b69e2d-e493-4ba2-9fb0-58a54d618ffc-kube-api-access-8dnq2\") pod \"network-check-source-65bdbbff45-sxq82\" (UID: \"70b69e2d-e493-4ba2-9fb0-58a54d618ffc\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" Aug 05 17:33:33.144788 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.144702 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2dt\" (UniqueName: \"kubernetes.io/projected/f7963633-3133-4b6d-af46-d0dd588a423c-kube-api-access-dm2dt\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.145896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.145847 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:33.218900 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.218866 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-nrs7p"] Aug 05 17:33:33.223283 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:33.223217 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be440ce_b9fe_486b_8edd_5fc930de9c3c.slice/crio-e94dd010a1b827ff826b6395625f14ccb46ce52470b6d765751a044b43807b0b WatchSource:0}: Error finding container e94dd010a1b827ff826b6395625f14ccb46ce52470b6d765751a044b43807b0b: Status 404 returned error can't find the container with id e94dd010a1b827ff826b6395625f14ccb46ce52470b6d765751a044b43807b0b Aug 05 17:33:33.228550 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.228468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" Aug 05 17:33:33.247280 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.247252 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:33.375110 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.375077 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djr4b" event={"ID":"cc290291-e2cb-42d0-a874-0d96b118fe28","Type":"ContainerStarted","Data":"d3ce5f1810c008f042f72953693af0ecdf9279003f884fbfa156241b319711cf"} Aug 05 17:33:33.376563 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.376536 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nntms" event={"ID":"817f6abd-e663-4560-b4d8-a7eaf8697446","Type":"ContainerStarted","Data":"ac84c8357494edd125335c6a9cdcc49909bb36a9466d0c9f1adef5494ff92ff0"} Aug 05 17:33:33.377750 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.377728 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" event={"ID":"1be440ce-b9fe-486b-8edd-5fc930de9c3c","Type":"ContainerStarted","Data":"e94dd010a1b827ff826b6395625f14ccb46ce52470b6d765751a044b43807b0b"} Aug 05 17:33:33.387704 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.387677 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82"] Aug 05 17:33:33.390922 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:33.390746 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b69e2d_e493_4ba2_9fb0_58a54d618ffc.slice/crio-0864d67365e7c4b7dfdbba749a2e08170eb204e1feef95087a3ec33b680f270f WatchSource:0}: Error finding container 0864d67365e7c4b7dfdbba749a2e08170eb204e1feef95087a3ec33b680f270f: Status 404 returned error can't find the container with id 0864d67365e7c4b7dfdbba749a2e08170eb204e1feef95087a3ec33b680f270f Aug 05 17:33:33.402432 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.402391 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-djr4b" podStartSLOduration=6.856573276 podStartE2EDuration="38.402376246s" totalImagesPullingTime="31.54580297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:33.402036825 +0000 UTC m=+39.878319291" watchObservedRunningTime="2026-08-05 17:33:33.402376246 +0000 UTC m=+39.878658710" Aug 05 17:33:33.422727 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.422684 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nntms" podStartSLOduration=2.041613527 podStartE2EDuration="4.422669789s" totalImagesPullingTime="2.381056262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:33.421787963 +0000 UTC m=+39.898070421" watchObservedRunningTime="2026-08-05 17:33:33.422669789 +0000 UTC m=+39.898952253" Aug 05 17:33:33.559762 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.559651 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:33.637221 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.637188 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.637286 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.637311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.637333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637336 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637356 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:33.637418 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637415 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:37.637395846 +0000 UTC m=+44.113678297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:33.637709 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637429 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:33:33.637709 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637438 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:33.637709 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637448 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:34.637433257 +0000 UTC m=+41.113715701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:33.637709 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637491 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:34.637476519 +0000 UTC m=+41.113758962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : secret "router-metrics-certs-default" not found Aug 05 17:33:33.637709 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:33.637503 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:37.637496329 +0000 UTC m=+44.113778778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:33.642870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.642846 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:33.739940 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:33.739895 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:34.382848 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.382803 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" event={"ID":"70b69e2d-e493-4ba2-9fb0-58a54d618ffc","Type":"ContainerStarted","Data":"0864d67365e7c4b7dfdbba749a2e08170eb204e1feef95087a3ec33b680f270f"} Aug 05 17:33:34.647094 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.647008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:34.647094 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.647074 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:34.647313 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:34.647195 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:33:34.647313 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:34.647271 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:36.647251613 +0000 UTC m=+43.123534060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:34.647313 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:34.647291 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:36.647282535 +0000 UTC m=+43.123564984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : secret "router-metrics-certs-default" not found Aug 05 17:33:34.658326 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.658292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:34.759576 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.759534 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:34.967116 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:34.967023 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:35.069617 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:35.069578 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:35.375802 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:35.375765 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" limit=1024 Aug 05 17:33:36.663545 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:36.663508 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:36.664016 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:36.663578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:36.664016 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:36.663695 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:33:36.664016 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:36.663744 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:40.66372438 +0000 UTC m=+47.140006823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:36.664016 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:36.663764 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:40.66375708 +0000 UTC m=+47.140039523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : secret "router-metrics-certs-default" not found Aug 05 17:33:37.395668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.392080 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/0.log" Aug 05 17:33:37.395668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.392434 2561 generic.go:356] "Generic (PLEG): container finished" podID="1be440ce-b9fe-486b-8edd-5fc930de9c3c" containerID="2422d5ff8c308557c58eaf166dd8e3daa2fac5053337b06f1bda060c5fe08bdd" exitCode=255 Aug 05 17:33:37.395668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.392988 2561 scope.go:117] "RemoveContainer" containerID="2422d5ff8c308557c58eaf166dd8e3daa2fac5053337b06f1bda060c5fe08bdd" Aug 05 17:33:37.395668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.393003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" event={"ID":"1be440ce-b9fe-486b-8edd-5fc930de9c3c","Type":"ContainerDied","Data":"2422d5ff8c308557c58eaf166dd8e3daa2fac5053337b06f1bda060c5fe08bdd"} Aug 05 17:33:37.397679 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.397648 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" event={"ID":"70b69e2d-e493-4ba2-9fb0-58a54d618ffc","Type":"ContainerStarted","Data":"9b477e31e2387ae6f49d944a5794b2dc8cb0ea6352f62ab3e8eb9a6835b70dba"} Aug 05 17:33:37.436250 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.436122 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-sxq82" podStartSLOduration=1.616585462 podStartE2EDuration="5.436104342s" totalImagesPullingTime="3.81951888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:37.435942317 +0000 UTC m=+43.912224829" watchObservedRunningTime="2026-08-05 17:33:37.436104342 +0000 UTC m=+43.912386807" Aug 05 17:33:37.672288 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.672253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:37.672343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:37.672368 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:37.672433 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:45.672412306 +0000 UTC m=+52.148694753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:37.672479 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:37.672496 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:37.672632 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:37.672548 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:45.672536335 +0000 UTC m=+52.148818778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:38.401687 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.401657 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:33:38.402067 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.402051 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/0.log" Aug 05 17:33:38.402178 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.402089 2561 generic.go:356] "Generic (PLEG): container finished" podID="1be440ce-b9fe-486b-8edd-5fc930de9c3c" containerID="c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb" exitCode=255 Aug 05 17:33:38.402252 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.402185 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" event={"ID":"1be440ce-b9fe-486b-8edd-5fc930de9c3c","Type":"ContainerDied","Data":"c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb"} Aug 05 17:33:38.402309 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.402263 2561 scope.go:117] "RemoveContainer" containerID="2422d5ff8c308557c58eaf166dd8e3daa2fac5053337b06f1bda060c5fe08bdd" Aug 05 17:33:38.402416 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:38.402389 2561 scope.go:117] "RemoveContainer" containerID="c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb" Aug 05 17:33:38.402582 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:38.402561 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-nrs7p_openshift-console-operator(1be440ce-b9fe-486b-8edd-5fc930de9c3c)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" podUID="1be440ce-b9fe-486b-8edd-5fc930de9c3c" Aug 05 17:33:39.405759 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.405725 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5cw"] Aug 05 17:33:39.406805 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.406788 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:33:39.407164 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.407150 2561 scope.go:117] "RemoveContainer" containerID="c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb" Aug 05 17:33:39.407350 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:39.407331 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-nrs7p_openshift-console-operator(1be440ce-b9fe-486b-8edd-5fc930de9c3c)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" podUID="1be440ce-b9fe-486b-8edd-5fc930de9c3c" Aug 05 17:33:39.432478 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.432446 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5cw"] Aug 05 17:33:39.432665 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.432592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.435618 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.435593 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 05 17:33:39.435794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.435772 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9t6s\"" Aug 05 17:33:39.435881 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.435780 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 05 17:33:39.435881 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.435803 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 05 17:33:39.435987 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.435918 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 05 17:33:39.586485 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.586447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scgh\" (UniqueName: \"kubernetes.io/projected/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-api-access-2scgh\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.586644 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.586539 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1207f1cd-50a0-4bbe-a218-004804dd5506-crio-socket\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.586644 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.586574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.586775 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.586743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.586921 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.586904 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1207f1cd-50a0-4bbe-a218-004804dd5506-data-volume\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.650331 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.650295 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq"] Aug 05 17:33:39.660394 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.660342 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" Aug 05 17:33:39.663568 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.663529 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hfspq\"" Aug 05 17:33:39.663884 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.663802 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 05 17:33:39.663884 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.663822 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 05 17:33:39.687352 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687471 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687379 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687471 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:39.687451 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:33:39.687558 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:39.687474 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:33:39.687558 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:39.687497 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:40.187483848 +0000 UTC m=+46.663766304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : configmap "kube-rbac-proxy" not found Aug 05 17:33:39.687558 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:39.687523 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:40.187509816 +0000 UTC m=+46.663792259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : secret "insights-runtime-extractor-tls" not found Aug 05 17:33:39.687689 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687576 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1207f1cd-50a0-4bbe-a218-004804dd5506-data-volume\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687689 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687604 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scgh\" (UniqueName: \"kubernetes.io/projected/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-api-access-2scgh\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687689 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1207f1cd-50a0-4bbe-a218-004804dd5506-crio-socket\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687890 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1207f1cd-50a0-4bbe-a218-004804dd5506-crio-socket\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.687963 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.687941 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1207f1cd-50a0-4bbe-a218-004804dd5506-data-volume\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.701544 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.701509 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq"] Aug 05 17:33:39.731218 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.731181 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scgh\" (UniqueName: \"kubernetes.io/projected/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-api-access-2scgh\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:39.788876 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.788843 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsxm\" (UniqueName: \"kubernetes.io/projected/837e03d2-d2e7-44e7-ab93-101a25cd3b18-kube-api-access-jzsxm\") pod \"migrator-68bc745d87-k7ntq\" (UID: \"837e03d2-d2e7-44e7-ab93-101a25cd3b18\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" Aug 05 17:33:39.889522 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.889478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsxm\" (UniqueName: \"kubernetes.io/projected/837e03d2-d2e7-44e7-ab93-101a25cd3b18-kube-api-access-jzsxm\") pod \"migrator-68bc745d87-k7ntq\" (UID: \"837e03d2-d2e7-44e7-ab93-101a25cd3b18\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" Aug 05 17:33:39.898464 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.898441 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsxm\" (UniqueName: \"kubernetes.io/projected/837e03d2-d2e7-44e7-ab93-101a25cd3b18-kube-api-access-jzsxm\") pod \"migrator-68bc745d87-k7ntq\" (UID: \"837e03d2-d2e7-44e7-ab93-101a25cd3b18\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" Aug 05 17:33:39.971604 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:39.971534 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" Aug 05 17:33:40.128711 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.128682 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq"] Aug 05 17:33:40.131565 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:40.131532 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837e03d2_d2e7_44e7_ab93_101a25cd3b18.slice/crio-5ba9e31050e7c6a4cf574aa540f85d2580e475620d87ceb723af8fe00e9a503e WatchSource:0}: Error finding container 5ba9e31050e7c6a4cf574aa540f85d2580e475620d87ceb723af8fe00e9a503e: Status 404 returned error can't find the container with id 5ba9e31050e7c6a4cf574aa540f85d2580e475620d87ceb723af8fe00e9a503e Aug 05 17:33:40.191277 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.191199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:40.191489 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.191290 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:40.191489 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.191389 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:33:40.191489 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.191403 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:33:40.191489 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.191471 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:41.191450345 +0000 UTC m=+47.667732788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : secret "insights-runtime-extractor-tls" not found Aug 05 17:33:40.191489 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.191492 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:41.19148229 +0000 UTC m=+47.667764733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : configmap "kube-rbac-proxy" not found Aug 05 17:33:40.410155 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.410114 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" event={"ID":"837e03d2-d2e7-44e7-ab93-101a25cd3b18","Type":"ContainerStarted","Data":"5ba9e31050e7c6a4cf574aa540f85d2580e475620d87ceb723af8fe00e9a503e"} Aug 05 17:33:40.695517 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.695417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:40.695517 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:40.695497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:40.695726 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.695570 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 05 17:33:40.695726 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.695631 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:48.695617777 +0000 UTC m=+55.171900221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : secret "router-metrics-certs-default" not found Aug 05 17:33:40.695726 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:40.695646 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:33:48.69564035 +0000 UTC m=+55.171922794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:41.171551 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:41.171520 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w965k_1382e3a7-4675-4b1c-a5a3-a193ab8f54a5/dns-node-resolver/0.log" Aug 05 17:33:41.200382 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:41.200347 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:41.200584 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:41.200483 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:33:41.200584 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:41.200506 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:41.200584 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:41.200559 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:43.200537991 +0000 UTC m=+49.676820448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : configmap "kube-rbac-proxy" not found Aug 05 17:33:41.200760 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:41.200621 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:33:41.200760 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:41.200673 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:43.200656463 +0000 UTC m=+49.676938910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : secret "insights-runtime-extractor-tls" not found Aug 05 17:33:42.166511 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:42.166486 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5t2w_e2436a4d-8c40-44f6-84c2-3c7c8663a80c/node-ca/0.log" Aug 05 17:33:42.417507 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:42.417475 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" event={"ID":"837e03d2-d2e7-44e7-ab93-101a25cd3b18","Type":"ContainerStarted","Data":"59ee37016c298a3622a595e0fe9c27bda97122080a035fe79d9468b8f7353f13"} Aug 05 17:33:42.417688 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:42.417524 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" event={"ID":"837e03d2-d2e7-44e7-ab93-101a25cd3b18","Type":"ContainerStarted","Data":"dcedeed75210f5089d2f932953815b373ad2fb54dd6180023dd90835b5c9c4ab"} Aug 05 17:33:42.767506 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:42.767473 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nntms_817f6abd-e663-4560-b4d8-a7eaf8697446/serve-healthcheck-canary/0.log" Aug 05 17:33:43.052572 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:43.052479 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:43.052572 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:43.052546 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:43.052859 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:43.052847 2561 scope.go:117] "RemoveContainer" containerID="c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb" Aug 05 17:33:43.053003 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:43.052985 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-nrs7p_openshift-console-operator(1be440ce-b9fe-486b-8edd-5fc930de9c3c)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" podUID="1be440ce-b9fe-486b-8edd-5fc930de9c3c" Aug 05 17:33:43.216258 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:43.216215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:43.216643 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:43.216277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:43.216643 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:43.216390 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:33:43.216643 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:43.216403 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 05 17:33:43.216643 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:43.216458 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:47.216437124 +0000 UTC m=+53.692719569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : secret "insights-runtime-extractor-tls" not found Aug 05 17:33:43.216643 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:43.216487 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:47.216475803 +0000 UTC m=+53.692758249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : configmap "kube-rbac-proxy" not found Aug 05 17:33:45.735604 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:45.735569 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:45.735632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:45.735727 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:45.735749 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:45.735759 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84c7947646-v9vcg: secret "image-registry-tls" not found Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:45.735797 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls podName:7de59047-10b9-441d-814d-2654e9f013a9 nodeName:}" failed. No retries permitted until 2026-08-05 17:34:01.735780549 +0000 UTC m=+68.212063012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls") pod "dns-default-nxfjd" (UID: "7de59047-10b9-441d-814d-2654e9f013a9") : secret "dns-default-metrics-tls" not found Aug 05 17:33:45.735991 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:45.735814 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls podName:f72f7646-7baa-483d-a6c1-4a23672593c3 nodeName:}" failed. No retries permitted until 2026-08-05 17:34:01.735806585 +0000 UTC m=+68.212089028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls") pod "image-registry-84c7947646-v9vcg" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3") : secret "image-registry-tls" not found Aug 05 17:33:47.246388 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:47.246352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:47.246844 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:47.246406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:47.246844 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:47.246508 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 05 17:33:47.246844 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:47.246570 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls podName:1207f1cd-50a0-4bbe-a218-004804dd5506 nodeName:}" failed. No retries permitted until 2026-08-05 17:33:55.246554528 +0000 UTC m=+61.722836972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rf5cw" (UID: "1207f1cd-50a0-4bbe-a218-004804dd5506") : secret "insights-runtime-extractor-tls" not found Aug 05 17:33:47.246985 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:47.246881 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1207f1cd-50a0-4bbe-a218-004804dd5506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:48.757474 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:48.757365 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:48.757474 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:48.757478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:48.757860 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:33:48.757554 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle podName:f7963633-3133-4b6d-af46-d0dd588a423c nodeName:}" failed. No retries permitted until 2026-08-05 17:34:04.75753287 +0000 UTC m=+71.233815317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle") pod "router-default-86984d778b-h5jmk" (UID: "f7963633-3133-4b6d-af46-d0dd588a423c") : configmap references non-existent config key: service-ca.crt Aug 05 17:33:48.759960 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:48.759928 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7963633-3133-4b6d-af46-d0dd588a423c-metrics-certs\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:33:55.312018 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.311981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:55.314633 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.314605 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1207f1cd-50a0-4bbe-a218-004804dd5506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rf5cw\" (UID: \"1207f1cd-50a0-4bbe-a218-004804dd5506\") " pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:55.343275 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.343250 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b9t6s\"" Aug 05 17:33:55.351291 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.351275 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rf5cw" Aug 05 17:33:55.356323 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.356301 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pv2lt" Aug 05 17:33:55.389073 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.389018 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-k7ntq" podStartSLOduration=14.440480869 podStartE2EDuration="16.388997965s" totalImagesPullingTime="1.948517096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:42.445978035 +0000 UTC m=+48.922260500" watchObservedRunningTime="2026-08-05 17:33:55.388997965 +0000 UTC m=+61.865280431" Aug 05 17:33:55.511272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:55.511220 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rf5cw"] Aug 05 17:33:55.515287 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:55.515255 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1207f1cd_50a0_4bbe_a218_004804dd5506.slice/crio-23fb17d7b011229bb5f18badb1af27128814e763f993ab929048e224f6ce6510 WatchSource:0}: Error finding container 23fb17d7b011229bb5f18badb1af27128814e763f993ab929048e224f6ce6510: Status 404 returned error can't find the container with id 23fb17d7b011229bb5f18badb1af27128814e763f993ab929048e224f6ce6510 Aug 05 17:33:56.452992 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:56.452958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5cw" event={"ID":"1207f1cd-50a0-4bbe-a218-004804dd5506","Type":"ContainerStarted","Data":"ba1f76d4c51b7a78a17de4c98ea61d5b7e8809bc7ca1d69ac45d49e56f29ade2"} Aug 05 17:33:56.453405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:56.453039 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5cw" event={"ID":"1207f1cd-50a0-4bbe-a218-004804dd5506","Type":"ContainerStarted","Data":"a4e547e9631a7348ca8091f26415ee7d4d2b77024dfaa2b161e823f044124158"} Aug 05 17:33:56.453405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:56.453058 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5cw" event={"ID":"1207f1cd-50a0-4bbe-a218-004804dd5506","Type":"ContainerStarted","Data":"23fb17d7b011229bb5f18badb1af27128814e763f993ab929048e224f6ce6510"} Aug 05 17:33:57.064195 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.064156 2561 scope.go:117] "RemoveContainer" containerID="c04c2d1c142dec3ec35d5145b383388ef8c6263baa3d4e590825fc14750272cb" Aug 05 17:33:57.458758 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.458684 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:33:57.458758 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.458738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" event={"ID":"1be440ce-b9fe-486b-8edd-5fc930de9c3c","Type":"ContainerStarted","Data":"e006dd0299b0be66f3973344266465aacdbc0ac029f6a2d959b70f589f46fe1a"} Aug 05 17:33:57.459259 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.459089 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:57.478652 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.478584 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" podStartSLOduration=22.915711267 podStartE2EDuration="25.478564845s" totalImagesPullingTime="2.562853578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:57.478373419 +0000 UTC m=+63.954655897" watchObservedRunningTime="2026-08-05 17:33:57.478564845 +0000 UTC m=+63.954847314" Aug 05 17:33:57.616181 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:57.616154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-nrs7p" Aug 05 17:33:58.464369 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:58.464333 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rf5cw" event={"ID":"1207f1cd-50a0-4bbe-a218-004804dd5506","Type":"ContainerStarted","Data":"34c0d3abb0230dd3b65f5f049df0c9b7c7dabc53d32bbadfac8f80cb8276f791"} Aug 05 17:33:58.488770 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:58.488721 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rf5cw" podStartSLOduration=17.510482624 podStartE2EDuration="19.488704689s" totalImagesPullingTime="1.978222065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:33:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:58.487875719 +0000 UTC m=+64.964158184" watchObservedRunningTime="2026-08-05 17:33:58.488704689 +0000 UTC m=+64.964987154" Aug 05 17:33:59.751334 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.751275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:59.751750 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.751350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:59.755195 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.755173 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 05 17:33:59.755356 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.755330 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 05 17:33:59.764714 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.764692 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a58be3a2-1580-4075-823e-05c7287b9b71-metrics-certs\") pod \"network-metrics-daemon-wcbrh\" (UID: \"a58be3a2-1580-4075-823e-05c7287b9b71\") " pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:59.764812 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.764780 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18f36257-69b8-428e-b68b-4913f2dad3cf-original-pull-secret\") pod \"global-pull-secret-syncer-4mfcd\" (UID: \"18f36257-69b8-428e-b68b-4913f2dad3cf\") " pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:59.779527 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.779503 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4mfcd" Aug 05 17:33:59.795477 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.795453 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bbv86\"" Aug 05 17:33:59.802824 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.802802 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcbrh" Aug 05 17:33:59.916639 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.916616 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4mfcd"] Aug 05 17:33:59.919340 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:59.919304 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f36257_69b8_428e_b68b_4913f2dad3cf.slice/crio-c0e3a4fc944c98277e6cabf1e653cf62a7d692fd3930bdfd6b673b4c2f172fe2 WatchSource:0}: Error finding container c0e3a4fc944c98277e6cabf1e653cf62a7d692fd3930bdfd6b673b4c2f172fe2: Status 404 returned error can't find the container with id c0e3a4fc944c98277e6cabf1e653cf62a7d692fd3930bdfd6b673b4c2f172fe2 Aug 05 17:33:59.942381 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.942363 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcbrh"] Aug 05 17:33:59.944442 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:33:59.944421 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58be3a2_1580_4075_823e_05c7287b9b71.slice/crio-a5a1ac4b97507e28f910a4a35cdc1995c3b52276fe45f4d01b3987f8b36468df WatchSource:0}: Error finding container a5a1ac4b97507e28f910a4a35cdc1995c3b52276fe45f4d01b3987f8b36468df: Status 404 returned error can't find the container with id a5a1ac4b97507e28f910a4a35cdc1995c3b52276fe45f4d01b3987f8b36468df Aug 05 17:33:59.953734 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.953711 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:33:59.956223 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:33:59.956204 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkfz\" (UniqueName: \"kubernetes.io/projected/ce729a7f-efb2-47bb-a2e2-d0877520c2b8-kube-api-access-tdkfz\") pod \"network-check-target-kdr45\" (UID: \"ce729a7f-efb2-47bb-a2e2-d0877520c2b8\") " pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:34:00.089729 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.089652 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dt589\"" Aug 05 17:34:00.097676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.097654 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:34:00.221657 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.221629 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kdr45"] Aug 05 17:34:00.224937 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:00.224910 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce729a7f_efb2_47bb_a2e2_d0877520c2b8.slice/crio-b553ca3982cbaeab09a31d5c7e197aec17edc055366423472d9d879b8e406412 WatchSource:0}: Error finding container b553ca3982cbaeab09a31d5c7e197aec17edc055366423472d9d879b8e406412: Status 404 returned error can't find the container with id b553ca3982cbaeab09a31d5c7e197aec17edc055366423472d9d879b8e406412 Aug 05 17:34:00.474188 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.474097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdr45" event={"ID":"ce729a7f-efb2-47bb-a2e2-d0877520c2b8","Type":"ContainerStarted","Data":"38e12dc827ada81a9fdb1cb776ecb3977c1265d75ea38099c17a7e4f6bcb8cf0"} Aug 05 17:34:00.474188 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.474149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kdr45" event={"ID":"ce729a7f-efb2-47bb-a2e2-d0877520c2b8","Type":"ContainerStarted","Data":"b553ca3982cbaeab09a31d5c7e197aec17edc055366423472d9d879b8e406412"} Aug 05 17:34:00.474188 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.474187 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:34:00.475664 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.475523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcbrh" event={"ID":"a58be3a2-1580-4075-823e-05c7287b9b71","Type":"ContainerStarted","Data":"a5a1ac4b97507e28f910a4a35cdc1995c3b52276fe45f4d01b3987f8b36468df"} Aug 05 17:34:00.477076 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.477046 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4mfcd" event={"ID":"18f36257-69b8-428e-b68b-4913f2dad3cf","Type":"ContainerStarted","Data":"c0e3a4fc944c98277e6cabf1e653cf62a7d692fd3930bdfd6b673b4c2f172fe2"} Aug 05 17:34:00.495708 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:00.495655 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kdr45" podStartSLOduration=65.495636341 podStartE2EDuration="1m5.495636341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:00.495472533 +0000 UTC m=+66.971755025" watchObservedRunningTime="2026-08-05 17:34:00.495636341 +0000 UTC m=+66.971918808" Aug 05 17:34:01.483090 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.482992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcbrh" event={"ID":"a58be3a2-1580-4075-823e-05c7287b9b71","Type":"ContainerStarted","Data":"ba014a4eaa9809bf611696da4526e122fa3867de1ab6ff4a0af340ad4fb9a48c"} Aug 05 17:34:01.483090 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.483060 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcbrh" event={"ID":"a58be3a2-1580-4075-823e-05c7287b9b71","Type":"ContainerStarted","Data":"e54ea0b6e8368ea1632c17b3c16d6079091033d4d782afb56e6629a036f0db09"} Aug 05 17:34:01.506295 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.506209 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wcbrh" podStartSLOduration=65.444417124 podStartE2EDuration="1m6.506189796s" totalImagesPullingTime="1.061772672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:01.505938435 +0000 UTC m=+67.982220900" watchObservedRunningTime="2026-08-05 17:34:01.506189796 +0000 UTC m=+67.982472262" Aug 05 17:34:01.768999 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.768966 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:34:01.769196 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.769017 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:01.771531 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.771502 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"image-registry-84c7947646-v9vcg\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:01.771531 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.771507 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7de59047-10b9-441d-814d-2654e9f013a9-metrics-tls\") pod \"dns-default-nxfjd\" (UID: \"7de59047-10b9-441d-814d-2654e9f013a9\") " pod="openshift-dns/dns-default-nxfjd" Aug 05 17:34:01.979157 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.979123 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jvd4r\"" Aug 05 17:34:01.986722 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.986687 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:01.992869 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:01.992836 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59qnw\"" Aug 05 17:34:02.000646 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.000623 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxfjd" Aug 05 17:34:02.167273 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.167111 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:34:02.169964 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:02.169938 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72f7646_7baa_483d_a6c1_4a23672593c3.slice/crio-5a58a9aa157cc33de7d9b3d7a1d973222bda354a64e2374a4add7699057d4044 WatchSource:0}: Error finding container 5a58a9aa157cc33de7d9b3d7a1d973222bda354a64e2374a4add7699057d4044: Status 404 returned error can't find the container with id 5a58a9aa157cc33de7d9b3d7a1d973222bda354a64e2374a4add7699057d4044 Aug 05 17:34:02.187731 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.187700 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxfjd"] Aug 05 17:34:02.192720 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:02.192685 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de59047_10b9_441d_814d_2654e9f013a9.slice/crio-467bb76188acd38fd67e6ded80f31df5d9aa02a0e9cd9027d1c3b9abec8d77ef WatchSource:0}: Error finding container 467bb76188acd38fd67e6ded80f31df5d9aa02a0e9cd9027d1c3b9abec8d77ef: Status 404 returned error can't find the container with id 467bb76188acd38fd67e6ded80f31df5d9aa02a0e9cd9027d1c3b9abec8d77ef Aug 05 17:34:02.488099 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.487999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxfjd" event={"ID":"7de59047-10b9-441d-814d-2654e9f013a9","Type":"ContainerStarted","Data":"467bb76188acd38fd67e6ded80f31df5d9aa02a0e9cd9027d1c3b9abec8d77ef"} Aug 05 17:34:02.489957 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.489923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" event={"ID":"f72f7646-7baa-483d-a6c1-4a23672593c3","Type":"ContainerStarted","Data":"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a"} Aug 05 17:34:02.490111 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.489980 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" event={"ID":"f72f7646-7baa-483d-a6c1-4a23672593c3","Type":"ContainerStarted","Data":"5a58a9aa157cc33de7d9b3d7a1d973222bda354a64e2374a4add7699057d4044"} Aug 05 17:34:02.514966 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:02.514117 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" podStartSLOduration=45.514093126 podStartE2EDuration="45.514093126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:02.51318474 +0000 UTC m=+68.989467206" watchObservedRunningTime="2026-08-05 17:34:02.514093126 +0000 UTC m=+68.990375593" Aug 05 17:34:03.494638 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:03.494602 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:04.499875 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:04.499839 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4mfcd" event={"ID":"18f36257-69b8-428e-b68b-4913f2dad3cf","Type":"ContainerStarted","Data":"25e8cedfe21310a2f7a1622a5b3ccab4c38d959f062a4c95b17078f55e050779"} Aug 05 17:34:04.515930 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:04.515872 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4mfcd" podStartSLOduration=65.264020488 podStartE2EDuration="1m9.515855419s" totalImagesPullingTime="4.251834931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:04.514663597 +0000 UTC m=+70.990946064" watchObservedRunningTime="2026-08-05 17:34:04.515855419 +0000 UTC m=+70.992137883" Aug 05 17:34:04.795132 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:04.795032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:04.796506 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:04.796477 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7963633-3133-4b6d-af46-d0dd588a423c-service-ca-bundle\") pod \"router-default-86984d778b-h5jmk\" (UID: \"f7963633-3133-4b6d-af46-d0dd588a423c\") " pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:05.048956 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:05.048878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7fvnv\"" Aug 05 17:34:05.057058 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:05.057031 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:05.199917 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:05.199884 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86984d778b-h5jmk"] Aug 05 17:34:05.537102 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:05.537071 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7963633_3133_4b6d_af46_d0dd588a423c.slice/crio-5534abf4af97d737a6aadd3d649f6ea0679cae59905ad779214687b27f30c1ec WatchSource:0}: Error finding container 5534abf4af97d737a6aadd3d649f6ea0679cae59905ad779214687b27f30c1ec: Status 404 returned error can't find the container with id 5534abf4af97d737a6aadd3d649f6ea0679cae59905ad779214687b27f30c1ec Aug 05 17:34:06.121179 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.121140 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29"] Aug 05 17:34:06.123211 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.123193 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.139506 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.139483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 05 17:34:06.140513 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.140492 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 05 17:34:06.140572 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.140501 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 05 17:34:06.140572 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.140500 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 05 17:34:06.171525 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.171497 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt"] Aug 05 17:34:06.173775 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.173750 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29"] Aug 05 17:34:06.173880 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.173864 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.179735 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:34:06.179707 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-143-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" Aug 05 17:34:06.179851 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.179789 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 05 17:34:06.180048 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.180027 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 05 17:34:06.180115 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.180055 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 05 17:34:06.206153 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.206122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06cefd7d-7e50-4563-b54b-5da839f2028e-tmp\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.206153 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.206155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgfk\" (UniqueName: \"kubernetes.io/projected/06cefd7d-7e50-4563-b54b-5da839f2028e-kube-api-access-ndgfk\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.206330 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.206179 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06cefd7d-7e50-4563-b54b-5da839f2028e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.221665 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.221638 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt"] Aug 05 17:34:06.307211 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307177 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdxm5\" (UniqueName: \"kubernetes.io/projected/2d512c1e-385b-4db1-a6da-a89d830643d5-kube-api-access-vdxm5\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307211 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307215 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307319 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307372 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06cefd7d-7e50-4563-b54b-5da839f2028e-tmp\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.307427 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgfk\" (UniqueName: \"kubernetes.io/projected/06cefd7d-7e50-4563-b54b-5da839f2028e-kube-api-access-ndgfk\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.307663 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06cefd7d-7e50-4563-b54b-5da839f2028e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.307663 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d512c1e-385b-4db1-a6da-a89d830643d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.307812 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.307793 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06cefd7d-7e50-4563-b54b-5da839f2028e-tmp\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.310042 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.310024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06cefd7d-7e50-4563-b54b-5da839f2028e-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.335283 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.335255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgfk\" (UniqueName: \"kubernetes.io/projected/06cefd7d-7e50-4563-b54b-5da839f2028e-kube-api-access-ndgfk\") pod \"klusterlet-addon-workmgr-7f55c4fd5f-26d29\" (UID: \"06cefd7d-7e50-4563-b54b-5da839f2028e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.408142 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408045 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d512c1e-385b-4db1-a6da-a89d830643d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408142 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdxm5\" (UniqueName: \"kubernetes.io/projected/2d512c1e-385b-4db1-a6da-a89d830643d5-kube-api-access-vdxm5\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408142 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408142 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408493 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408493 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408182 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.408860 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.408836 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d512c1e-385b-4db1-a6da-a89d830643d5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.410820 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.410798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.410901 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.410818 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-ca\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.410901 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.410894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.421871 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.421842 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdxm5\" (UniqueName: \"kubernetes.io/projected/2d512c1e-385b-4db1-a6da-a89d830643d5-kube-api-access-vdxm5\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:06.431870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.431835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:06.508778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.508740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxfjd" event={"ID":"7de59047-10b9-441d-814d-2654e9f013a9","Type":"ContainerStarted","Data":"58d8d22541834839d6655cfd336bd9543cf24cde07c63c2cb82d685d57b1aba7"} Aug 05 17:34:06.508973 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.508818 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nxfjd" Aug 05 17:34:06.508973 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.508846 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxfjd" event={"ID":"7de59047-10b9-441d-814d-2654e9f013a9","Type":"ContainerStarted","Data":"5692d7feb1cd2fecda953ec5007fdcd591dd7e834267e7f0bc11a930f22045cd"} Aug 05 17:34:06.510218 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.510188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86984d778b-h5jmk" event={"ID":"f7963633-3133-4b6d-af46-d0dd588a423c","Type":"ContainerStarted","Data":"986eb4a2a7a45d922f756b57ab28611dfce528582eb01d2cc3a0ed82223bc0ae"} Aug 05 17:34:06.510342 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.510273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86984d778b-h5jmk" event={"ID":"f7963633-3133-4b6d-af46-d0dd588a423c","Type":"ContainerStarted","Data":"5534abf4af97d737a6aadd3d649f6ea0679cae59905ad779214687b27f30c1ec"} Aug 05 17:34:06.536447 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.536387 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nxfjd" podStartSLOduration=34.164972331 podStartE2EDuration="37.536359878s" totalImagesPullingTime="3.371387547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:06.535477099 +0000 UTC m=+73.011759560" watchObservedRunningTime="2026-08-05 17:34:06.536359878 +0000 UTC m=+73.012642342" Aug 05 17:34:06.562043 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.561986 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86984d778b-h5jmk" podStartSLOduration=34.561963955 podStartE2EDuration="34.561963955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:06.561819842 +0000 UTC m=+73.038102319" watchObservedRunningTime="2026-08-05 17:34:06.561963955 +0000 UTC m=+73.038246420" Aug 05 17:34:06.571108 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:06.571079 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29"] Aug 05 17:34:06.574845 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:06.574816 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cefd7d_7e50_4563_b54b_5da839f2028e.slice/crio-27ae6b8b3e4417f6b9d558cb1d60970d22c675e67ad67ab3da336a80ec3e22e8 WatchSource:0}: Error finding container 27ae6b8b3e4417f6b9d558cb1d60970d22c675e67ad67ab3da336a80ec3e22e8: Status 404 returned error can't find the container with id 27ae6b8b3e4417f6b9d558cb1d60970d22c675e67ad67ab3da336a80ec3e22e8 Aug 05 17:34:07.057875 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.057833 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:07.060256 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.060211 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:07.368291 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.368179 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 05 17:34:07.370759 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.370738 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d512c1e-385b-4db1-a6da-a89d830643d5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt\" (UID: \"2d512c1e-385b-4db1-a6da-a89d830643d5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:07.399871 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.399846 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" Aug 05 17:34:07.517351 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.517303 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" event={"ID":"06cefd7d-7e50-4563-b54b-5da839f2028e","Type":"ContainerStarted","Data":"27ae6b8b3e4417f6b9d558cb1d60970d22c675e67ad67ab3da336a80ec3e22e8"} Aug 05 17:34:07.517556 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.517538 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:07.520627 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.520438 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86984d778b-h5jmk" Aug 05 17:34:07.573510 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:07.573474 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt"] Aug 05 17:34:07.580463 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:07.580433 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d512c1e_385b_4db1_a6da_a89d830643d5.slice/crio-7627062e20c1fd6e0d1c9b6dee64bf0f5ac5ec3d4b66ef460ee7803ae2f0b889 WatchSource:0}: Error finding container 7627062e20c1fd6e0d1c9b6dee64bf0f5ac5ec3d4b66ef460ee7803ae2f0b889: Status 404 returned error can't find the container with id 7627062e20c1fd6e0d1c9b6dee64bf0f5ac5ec3d4b66ef460ee7803ae2f0b889 Aug 05 17:34:08.522700 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:08.522629 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" event={"ID":"2d512c1e-385b-4db1-a6da-a89d830643d5","Type":"ContainerStarted","Data":"7627062e20c1fd6e0d1c9b6dee64bf0f5ac5ec3d4b66ef460ee7803ae2f0b889"} Aug 05 17:34:13.544000 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.543963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" event={"ID":"06cefd7d-7e50-4563-b54b-5da839f2028e","Type":"ContainerStarted","Data":"603be6d6de3c20a866ca1532c928290b3f2207dcb44df1a71990eab84afde341"} Aug 05 17:34:13.544477 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.544133 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:13.546209 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.546180 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" Aug 05 17:34:13.546455 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.546432 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" event={"ID":"2d512c1e-385b-4db1-a6da-a89d830643d5","Type":"ContainerStarted","Data":"dbf80afe7c915431f910f056c478f4b534737fa90b414e78b49cab7ed30270ba"} Aug 05 17:34:13.546540 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.546471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" event={"ID":"2d512c1e-385b-4db1-a6da-a89d830643d5","Type":"ContainerStarted","Data":"e10fe74c24d6cc7284cba99f052e75a3f8e2cec2e17e3fac071df388008f80c6"} Aug 05 17:34:13.546540 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.546490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" event={"ID":"2d512c1e-385b-4db1-a6da-a89d830643d5","Type":"ContainerStarted","Data":"0f37b99d7316ce6fd3c6fa24181cc249b9a388ddfde9bc357fe4b1b9bcdd07d9"} Aug 05 17:34:13.564163 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.564108 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f55c4fd5f-26d29" podStartSLOduration=1.142689039 podStartE2EDuration="7.564091877s" totalImagesPullingTime="6.421402838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:13.562858682 +0000 UTC m=+80.039141147" watchObservedRunningTime="2026-08-05 17:34:13.564091877 +0000 UTC m=+80.040374342" Aug 05 17:34:13.607264 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:13.607165 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b58cb9bc6-c9jbt" podStartSLOduration=2.174154659 podStartE2EDuration="7.607143204s" totalImagesPullingTime="5.432988545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:13.604184162 +0000 UTC m=+80.080466667" watchObservedRunningTime="2026-08-05 17:34:13.607143204 +0000 UTC m=+80.083425671" Aug 05 17:34:16.519459 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:16.519347 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nxfjd" Aug 05 17:34:17.570247 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.570186 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hkl6c"] Aug 05 17:34:17.574883 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.574854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.578141 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578106 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 05 17:34:17.578314 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578144 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 05 17:34:17.578314 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578205 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 05 17:34:17.578883 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578866 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 05 17:34:17.578985 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578966 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p8psp\"" Aug 05 17:34:17.579048 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.578980 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 05 17:34:17.579097 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.579077 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 05 17:34:17.701192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701154 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-sys\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701192 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701202 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-root\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701426 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701290 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-textfile\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701426 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701383 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-tls\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701426 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701423 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701526 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701446 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701526 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701467 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6g5\" (UniqueName: \"kubernetes.io/projected/5c83618d-c200-409b-ba0d-9bb3b332a4ef-kube-api-access-9j6g5\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701526 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701498 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-wtmp\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.701619 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.701554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-metrics-client-ca\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802682 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-sys\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802682 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-root\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-textfile\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-tls\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802758 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-sys\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-root\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802818 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.802927 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802898 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6g5\" (UniqueName: \"kubernetes.io/projected/5c83618d-c200-409b-ba0d-9bb3b332a4ef-kube-api-access-9j6g5\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802932 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-wtmp\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.802982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-metrics-client-ca\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.803120 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-textfile\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803272 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.803162 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-wtmp\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803493 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.803431 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.803550 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.803520 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c83618d-c200-409b-ba0d-9bb3b332a4ef-metrics-client-ca\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.805409 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.805379 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.805540 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.805521 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c83618d-c200-409b-ba0d-9bb3b332a4ef-node-exporter-tls\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.812373 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.812345 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6g5\" (UniqueName: \"kubernetes.io/projected/5c83618d-c200-409b-ba0d-9bb3b332a4ef-kube-api-access-9j6g5\") pod \"node-exporter-hkl6c\" (UID: \"5c83618d-c200-409b-ba0d-9bb3b332a4ef\") " pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.885256 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:17.885156 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hkl6c" Aug 05 17:34:17.896715 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:34:17.896683 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c83618d_c200_409b_ba0d_9bb3b332a4ef.slice/crio-8676ca5651c9ea7be0476c7cc68b64f93a62421fe8bebc5eda8f11b9d792fb3d WatchSource:0}: Error finding container 8676ca5651c9ea7be0476c7cc68b64f93a62421fe8bebc5eda8f11b9d792fb3d: Status 404 returned error can't find the container with id 8676ca5651c9ea7be0476c7cc68b64f93a62421fe8bebc5eda8f11b9d792fb3d Aug 05 17:34:18.571286 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:18.571251 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkl6c" event={"ID":"5c83618d-c200-409b-ba0d-9bb3b332a4ef","Type":"ContainerStarted","Data":"8676ca5651c9ea7be0476c7cc68b64f93a62421fe8bebc5eda8f11b9d792fb3d"} Aug 05 17:34:19.575770 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:19.575736 2561 generic.go:356] "Generic (PLEG): container finished" podID="5c83618d-c200-409b-ba0d-9bb3b332a4ef" containerID="ada36779d58a7a02af3d35c4bca8e361bb778900268ed628bfe19772f220118b" exitCode=0 Aug 05 17:34:19.576135 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:19.575787 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkl6c" event={"ID":"5c83618d-c200-409b-ba0d-9bb3b332a4ef","Type":"ContainerDied","Data":"ada36779d58a7a02af3d35c4bca8e361bb778900268ed628bfe19772f220118b"} Aug 05 17:34:20.581149 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:20.581113 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkl6c" event={"ID":"5c83618d-c200-409b-ba0d-9bb3b332a4ef","Type":"ContainerStarted","Data":"0360cdf78e85fc2481efbe53f8374ff5fa17ce87cd433db1eb6538bf50020fd7"} Aug 05 17:34:20.581617 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:20.581164 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hkl6c" event={"ID":"5c83618d-c200-409b-ba0d-9bb3b332a4ef","Type":"ContainerStarted","Data":"92310709ec7f6bade61461e555783e65b24da05e388f5da0eca4a46aee930805"} Aug 05 17:34:20.604336 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:20.604290 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hkl6c" podStartSLOduration=2.481117301 podStartE2EDuration="3.60427268s" totalImagesPullingTime="1.050681679s" totalInitContainerRuntime="72.4737ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:20.604003826 +0000 UTC m=+87.080286291" watchObservedRunningTime="2026-08-05 17:34:20.60427268 +0000 UTC m=+87.080555145" Aug 05 17:34:21.991156 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:21.991105 2561 patch_prober.go:28] interesting pod/image-registry-84c7947646-v9vcg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 05 17:34:21.991691 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:21.991171 2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 17:34:24.504449 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:24.504414 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:29.773976 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:29.773937 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:34:31.485646 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:31.485613 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kdr45" Aug 05 17:34:54.793709 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:54.793652 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerName="registry" containerID="cri-o://9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a" gracePeriod=30 Aug 05 17:34:55.040687 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.040654 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:55.079537 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079464 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079537 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079509 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg9pv\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079743 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079540 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079743 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079589 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079743 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079609 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079865 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079779 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079865 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079850 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.079957 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.079892 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca\") pod \"f72f7646-7baa-483d-a6c1-4a23672593c3\" (UID: \"f72f7646-7baa-483d-a6c1-4a23672593c3\") " Aug 05 17:34:55.080535 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.080353 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:34:55.080668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.080648 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 17:34:55.082515 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.082470 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:34:55.082672 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.082573 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:34:55.082784 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.082684 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:34:55.082827 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.082775 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:34:55.083094 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.083073 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv" (OuterVolumeSpecName: "kube-api-access-sg9pv") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "kube-api-access-sg9pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:34:55.089094 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.089063 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f72f7646-7baa-483d-a6c1-4a23672593c3" (UID: "f72f7646-7baa-483d-a6c1-4a23672593c3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:34:55.180473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180438 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f72f7646-7baa-483d-a6c1-4a23672593c3-ca-trust-extracted\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180473 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180470 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-bound-sa-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180485 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-image-registry-private-configuration\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180494 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-certificates\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180503 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f7646-7baa-483d-a6c1-4a23672593c3-trusted-ca\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180512 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f72f7646-7baa-483d-a6c1-4a23672593c3-installation-pull-secrets\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180519 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9pv\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-kube-api-access-sg9pv\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.180686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.180527 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f72f7646-7baa-483d-a6c1-4a23672593c3-registry-tls\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:34:55.708869 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.708833 2561 generic.go:356] "Generic (PLEG): container finished" podID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerID="9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a" exitCode=0 Aug 05 17:34:55.709054 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.708883 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" Aug 05 17:34:55.709054 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.708921 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" event={"ID":"f72f7646-7baa-483d-a6c1-4a23672593c3","Type":"ContainerDied","Data":"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a"} Aug 05 17:34:55.709054 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.708974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84c7947646-v9vcg" event={"ID":"f72f7646-7baa-483d-a6c1-4a23672593c3","Type":"ContainerDied","Data":"5a58a9aa157cc33de7d9b3d7a1d973222bda354a64e2374a4add7699057d4044"} Aug 05 17:34:55.709054 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.708999 2561 scope.go:117] "RemoveContainer" containerID="9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a" Aug 05 17:34:55.717402 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.717389 2561 scope.go:117] "RemoveContainer" containerID="9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a" Aug 05 17:34:55.717653 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:34:55.717630 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a\": container with ID starting with 9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a not found: ID does not exist" containerID="9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a" Aug 05 17:34:55.717709 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.717662 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a"} err="failed to get container status \"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a\": rpc error: code = NotFound desc = could not find container \"9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a\": container with ID starting with 9243c5368322f7b23f59a67ff3e5c017eded45405349328f7e092171462c768a not found: ID does not exist" Aug 05 17:34:55.729450 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.729429 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:34:55.732841 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:55.732821 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84c7947646-v9vcg"] Aug 05 17:34:56.067930 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:34:56.067896 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" path="/var/lib/kubelet/pods/f72f7646-7baa-483d-a6c1-4a23672593c3/volumes" Aug 05 17:37:53.925575 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:37:53.925546 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:37:53.927256 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:37:53.927217 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:37:53.935965 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:37:53.935939 2561 kubelet.go:1648] "Image garbage collection succeeded" Aug 05 17:38:59.827271 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.827218 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq"] Aug 05 17:38:59.827695 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.827511 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerName="registry" Aug 05 17:38:59.827695 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.827522 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerName="registry" Aug 05 17:38:59.827695 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.827565 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72f7646-7baa-483d-a6c1-4a23672593c3" containerName="registry" Aug 05 17:38:59.829275 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.829258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:38:59.833788 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.833751 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 05 17:38:59.833966 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.833816 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 05 17:38:59.834629 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.834608 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 05 17:38:59.834733 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.834634 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 05 17:38:59.835007 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.834985 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hqhjq\"" Aug 05 17:38:59.836429 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.836386 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 05 17:38:59.848736 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.848707 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq"] Aug 05 17:38:59.936128 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.936084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:38:59.936128 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.936128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6wv\" (UniqueName: \"kubernetes.io/projected/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-kube-api-access-hl6wv\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:38:59.936410 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.936152 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-metrics-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:38:59.936410 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:38:59.936266 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-manager-config\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.037454 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.037417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-manager-config\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.037581 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.037490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.037581 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.037511 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6wv\" (UniqueName: \"kubernetes.io/projected/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-kube-api-access-hl6wv\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.037581 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.037529 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-metrics-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.038150 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.038123 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-manager-config\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.040166 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.040133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-metrics-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.040298 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.040169 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-cert\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.049828 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.049795 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6wv\" (UniqueName: \"kubernetes.io/projected/d63a9e85-c233-4dcb-ab8f-91196f38a0f2-kube-api-access-hl6wv\") pod \"lws-controller-manager-dd9d7cdcd-cpxnq\" (UID: \"d63a9e85-c233-4dcb-ab8f-91196f38a0f2\") " pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.139013 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.138930 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:00.288929 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.288889 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq"] Aug 05 17:39:00.291892 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:39:00.291863 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63a9e85_c233_4dcb_ab8f_91196f38a0f2.slice/crio-4e6c069cf3216d38fd0fda93163c20aeb3969e17f9eba162eadff2baeb1d1a53 WatchSource:0}: Error finding container 4e6c069cf3216d38fd0fda93163c20aeb3969e17f9eba162eadff2baeb1d1a53: Status 404 returned error can't find the container with id 4e6c069cf3216d38fd0fda93163c20aeb3969e17f9eba162eadff2baeb1d1a53 Aug 05 17:39:00.293787 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.293770 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:39:00.522387 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:00.522346 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" event={"ID":"d63a9e85-c233-4dcb-ab8f-91196f38a0f2","Type":"ContainerStarted","Data":"4e6c069cf3216d38fd0fda93163c20aeb3969e17f9eba162eadff2baeb1d1a53"} Aug 05 17:39:01.364264 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.364201 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z"] Aug 05 17:39:01.367322 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.367276 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.370782 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.370728 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 05 17:39:01.371121 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.370961 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 05 17:39:01.371121 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.370991 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xmkgp\"" Aug 05 17:39:01.371121 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.371075 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 05 17:39:01.371364 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.370963 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 05 17:39:01.392123 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.392041 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z"] Aug 05 17:39:01.449745 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.449704 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-manifests\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.449935 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.449754 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.449935 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.449866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-tmp\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.449935 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.449901 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-webhook-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.450109 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.449984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlcm\" (UniqueName: \"kubernetes.io/projected/d505ec41-5568-415d-b0f2-ba6fbd311339-kube-api-access-8nlcm\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.550601 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.550560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.550601 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.550607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-tmp\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.550858 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.550639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-webhook-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.550858 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.550668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlcm\" (UniqueName: \"kubernetes.io/projected/d505ec41-5568-415d-b0f2-ba6fbd311339-kube-api-access-8nlcm\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.550858 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.550763 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-manifests\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.551197 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.551152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-manifests\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.553515 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.553490 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d505ec41-5568-415d-b0f2-ba6fbd311339-tmp\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.553798 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.553779 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.553850 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.553833 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d505ec41-5568-415d-b0f2-ba6fbd311339-webhook-cert\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.563061 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.563038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlcm\" (UniqueName: \"kubernetes.io/projected/d505ec41-5568-415d-b0f2-ba6fbd311339-kube-api-access-8nlcm\") pod \"opendatahub-operator-controller-manager-856dccdfcf-6cn8z\" (UID: \"d505ec41-5568-415d-b0f2-ba6fbd311339\") " pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:01.682060 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:01.681971 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:02.802806 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:02.802781 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z"] Aug 05 17:39:02.805515 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:39:02.805489 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd505ec41_5568_415d_b0f2_ba6fbd311339.slice/crio-8caed5d1903434dd05f348a30cf915f2e818f31ac95be39f5fd82c2034fe3a6a WatchSource:0}: Error finding container 8caed5d1903434dd05f348a30cf915f2e818f31ac95be39f5fd82c2034fe3a6a: Status 404 returned error can't find the container with id 8caed5d1903434dd05f348a30cf915f2e818f31ac95be39f5fd82c2034fe3a6a Aug 05 17:39:03.535703 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:03.535660 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" event={"ID":"d505ec41-5568-415d-b0f2-ba6fbd311339","Type":"ContainerStarted","Data":"8caed5d1903434dd05f348a30cf915f2e818f31ac95be39f5fd82c2034fe3a6a"} Aug 05 17:39:03.537699 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:03.537619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" event={"ID":"d63a9e85-c233-4dcb-ab8f-91196f38a0f2","Type":"ContainerStarted","Data":"a122c8bc1c15116fa420e3b1b9bd9f7000a74eee9d7e59c488d366024fed43de"} Aug 05 17:39:03.537861 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:03.537823 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:03.562080 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:03.562023 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" podStartSLOduration=2.159420793 podStartE2EDuration="4.562006496s" totalImagesPullingTime="2.402585703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:38:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:03.560942026 +0000 UTC m=+370.037224492" watchObservedRunningTime="2026-08-05 17:39:03.562006496 +0000 UTC m=+370.038288963" Aug 05 17:39:06.550833 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:06.550790 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" event={"ID":"d505ec41-5568-415d-b0f2-ba6fbd311339","Type":"ContainerStarted","Data":"3317c0cb8b14559e33687c4f2ce47727cd32695dce06f5e47bf325bb2cc6a3a3"} Aug 05 17:39:07.556105 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:07.556065 2561 generic.go:356] "Generic (PLEG): container finished" podID="d505ec41-5568-415d-b0f2-ba6fbd311339" containerID="3317c0cb8b14559e33687c4f2ce47727cd32695dce06f5e47bf325bb2cc6a3a3" exitCode=0 Aug 05 17:39:07.556105 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:07.556111 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" event={"ID":"d505ec41-5568-415d-b0f2-ba6fbd311339","Type":"ContainerDied","Data":"3317c0cb8b14559e33687c4f2ce47727cd32695dce06f5e47bf325bb2cc6a3a3"} Aug 05 17:39:08.562048 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:08.562004 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" event={"ID":"d505ec41-5568-415d-b0f2-ba6fbd311339","Type":"ContainerStarted","Data":"0dd27f8618d034dc58a5d1f5f4b7461330588cd1fddb421090aca29f43ead42a"} Aug 05 17:39:08.562510 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:08.562154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:08.589802 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:08.589734 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" podStartSLOduration=2.881419069 podStartE2EDuration="7.589610738s" totalImagesPullingTime="3.699667879s" totalInitContainerRuntime="1.00852379s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:39:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:08.588466057 +0000 UTC m=+375.064748521" watchObservedRunningTime="2026-08-05 17:39:08.589610738 +0000 UTC m=+375.065893206" Aug 05 17:39:14.543918 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:14.543885 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-dd9d7cdcd-cpxnq" Aug 05 17:39:19.568742 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:19.568706 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856dccdfcf-6cn8z" Aug 05 17:39:31.727406 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.727371 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k"] Aug 05 17:39:31.735809 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.735787 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.738340 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.738314 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 05 17:39:31.739881 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.739866 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 05 17:39:31.740072 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.740057 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 05 17:39:31.740173 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.740153 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9rhln\"" Aug 05 17:39:31.740569 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.740555 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 05 17:39:31.745987 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.745960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k"] Aug 05 17:39:31.890545 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890502 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890545 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890543 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tgt\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-kube-api-access-d6tgt\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890700 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02c55d1-d6d4-4ae2-9427-52c32f294adb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.890778 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.890743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.991794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991689 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.991794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02c55d1-d6d4-4ae2-9427-52c32f294adb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.991794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.991794 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.992133 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.992133 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991899 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.992133 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.991937 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tgt\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-kube-api-access-d6tgt\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.992562 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.992532 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.994413 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.994384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02c55d1-d6d4-4ae2-9427-52c32f294adb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.994682 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.994666 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.994754 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.994694 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:31.994827 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:31.994791 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:32.006278 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:32.006213 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tgt\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-kube-api-access-d6tgt\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:32.007291 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:32.007269 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02c55d1-d6d4-4ae2-9427-52c32f294adb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lpf2k\" (UID: \"c02c55d1-d6d4-4ae2-9427-52c32f294adb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:32.045440 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:32.045404 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:32.278301 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:32.278273 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k"] Aug 05 17:39:32.279185 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:39:32.279157 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02c55d1_d6d4_4ae2_9427_52c32f294adb.slice/crio-1d3132c824b3dc146bcf63447ee9d5c6b4eeb07a03e9992db59af0cb802c469d WatchSource:0}: Error finding container 1d3132c824b3dc146bcf63447ee9d5c6b4eeb07a03e9992db59af0cb802c469d: Status 404 returned error can't find the container with id 1d3132c824b3dc146bcf63447ee9d5c6b4eeb07a03e9992db59af0cb802c469d Aug 05 17:39:32.650993 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:32.650908 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" event={"ID":"c02c55d1-d6d4-4ae2-9427-52c32f294adb","Type":"ContainerStarted","Data":"1d3132c824b3dc146bcf63447ee9d5c6b4eeb07a03e9992db59af0cb802c469d"} Aug 05 17:39:38.303153 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:38.303108 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:39:38.303512 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:38.303183 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 17:39:38.674731 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:38.674637 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" event={"ID":"c02c55d1-d6d4-4ae2-9427-52c32f294adb","Type":"ContainerStarted","Data":"3c018f40dec6eb8d43c3f48ec8fc93263f3fb7b8d6d653c8af7eec79dc912cf2"} Aug 05 17:39:38.674896 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:38.674771 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:39:38.702490 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:38.702441 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" podStartSLOduration=1.680794213 podStartE2EDuration="7.702425848s" totalImagesPullingTime="6.021631635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:38.70083624 +0000 UTC m=+405.177118701" watchObservedRunningTime="2026-08-05 17:39:38.702425848 +0000 UTC m=+405.178708314" Aug 05 17:39:39.680668 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:39:39.680640 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lpf2k" Aug 05 17:41:11.216286 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.216248 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th"] Aug 05 17:41:11.218208 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.218191 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.221098 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.221077 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 05 17:41:11.221222 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.221125 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hfkjp\"" Aug 05 17:41:11.221407 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.221393 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 05 17:41:11.232578 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.232554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th"] Aug 05 17:41:11.259587 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.259558 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.259587 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.259593 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dbz\" (UniqueName: \"kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.285088 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.285055 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th"] Aug 05 17:41:11.285334 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:11.285315 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-p9dbz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" Aug 05 17:41:11.294851 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.294820 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th"] Aug 05 17:41:11.360403 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.360362 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.360403 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.360400 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dbz\" (UniqueName: \"kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.360792 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.360771 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.362502 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:11.362484 2561 projected.go:196] Error preparing data for projected volume kube-api-access-p9dbz for pod kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-143-87.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object Aug 05 17:41:11.362607 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:11.362559 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz podName:1b62f2af-643c-4652-9cb2-87158dc41ebf nodeName:}" failed. No retries permitted until 2026-08-05 17:41:11.862541409 +0000 UTC m=+498.338823854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p9dbz" (UniqueName: "kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz") pod "kuadrant-operator-controller-manager-68cf589589-jx2th" (UID: "1b62f2af-643c-4652-9cb2-87158dc41ebf") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-143-87.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object Aug 05 17:41:11.863720 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:11.863683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dbz\" (UniqueName: \"kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz\") pod \"kuadrant-operator-controller-manager-68cf589589-jx2th\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:11.866296 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:11.866276 2561 projected.go:196] Error preparing data for projected volume kube-api-access-p9dbz for pod kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-143-87.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object Aug 05 17:41:11.866378 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:11.866342 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz podName:1b62f2af-643c-4652-9cb2-87158dc41ebf nodeName:}" failed. No retries permitted until 2026-08-05 17:41:12.866322014 +0000 UTC m=+499.342604474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p9dbz" (UniqueName: "kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz") pod "kuadrant-operator-controller-manager-68cf589589-jx2th" (UID: "1b62f2af-643c-4652-9cb2-87158dc41ebf") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-143-87.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object Aug 05 17:41:12.002089 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.002057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:12.004494 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:12.004467 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-jx2th\" is forbidden: User \"system:node:ip-10-0-143-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:12.007260 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.007218 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:12.009487 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:12.009463 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-jx2th\" is forbidden: User \"system:node:ip-10-0-143-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:12.064835 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.064804 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dbz\" (UniqueName: \"kubernetes.io/projected/1b62f2af-643c-4652-9cb2-87158dc41ebf-kube-api-access-p9dbz\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:41:12.165986 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.165900 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume\") pod \"1b62f2af-643c-4652-9cb2-87158dc41ebf\" (UID: \"1b62f2af-643c-4652-9cb2-87158dc41ebf\") " Aug 05 17:41:12.166277 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.166215 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b62f2af-643c-4652-9cb2-87158dc41ebf" (UID: "1b62f2af-643c-4652-9cb2-87158dc41ebf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:41:12.266479 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:12.266443 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b62f2af-643c-4652-9cb2-87158dc41ebf-extensions-socket-volume\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:41:13.005589 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:13.005549 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:13.013481 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:13.013448 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-jx2th\" is forbidden: User \"system:node:ip-10-0-143-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:14.067730 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:41:14.067686 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-jx2th\" is forbidden: User \"system:node:ip-10-0-143-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-87.ec2.internal' and this object" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-jx2th" Aug 05 17:41:14.068257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:14.068210 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b62f2af-643c-4652-9cb2-87158dc41ebf" path="/var/lib/kubelet/pods/1b62f2af-643c-4652-9cb2-87158dc41ebf/volumes" Aug 05 17:41:45.297465 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.297377 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:41:45.299714 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.299697 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.302182 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.302154 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 05 17:41:45.302322 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.302273 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 05 17:41:45.302849 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.302835 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 05 17:41:45.302885 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.302846 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vfp92\"" Aug 05 17:41:45.309388 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.309363 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:41:45.343730 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.343693 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:41:45.429862 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.429811 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-config-file\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.430045 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.429881 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gft\" (UniqueName: \"kubernetes.io/projected/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-kube-api-access-s5gft\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.530805 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.530767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gft\" (UniqueName: \"kubernetes.io/projected/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-kube-api-access-s5gft\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.530981 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.530816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-config-file\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.531467 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.531447 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-config-file\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.539612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.539594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gft\" (UniqueName: \"kubernetes.io/projected/ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6-kube-api-access-s5gft\") pod \"limitador-limitador-85d9db4d5d-hcw6v\" (UID: \"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.610657 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.610569 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:45.741474 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:45.741442 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:41:45.745157 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:41:45.745126 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4cccd0_356c_409f_acf0_ff9ef1a0d6c6.slice/crio-f9727178f7016d2c059d1c62e82b7fcb2a97897e52428423bf652afbf49fafb9 WatchSource:0}: Error finding container f9727178f7016d2c059d1c62e82b7fcb2a97897e52428423bf652afbf49fafb9: Status 404 returned error can't find the container with id f9727178f7016d2c059d1c62e82b7fcb2a97897e52428423bf652afbf49fafb9 Aug 05 17:41:46.124556 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:46.124521 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" event={"ID":"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6","Type":"ContainerStarted","Data":"f9727178f7016d2c059d1c62e82b7fcb2a97897e52428423bf652afbf49fafb9"} Aug 05 17:41:49.138780 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:49.138742 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" event={"ID":"ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6","Type":"ContainerStarted","Data":"ef44c619a621b936bcbf509d899a6fc67f7a2c100e7ba15504f7a4ae22dfdcc7"} Aug 05 17:41:49.139160 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:49.138886 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:41:49.155930 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:41:49.155876 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" podStartSLOduration=1.735576528 podStartE2EDuration="4.155858621s" totalImagesPullingTime="2.420282093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:41:49.155754466 +0000 UTC m=+535.632036933" watchObservedRunningTime="2026-08-05 17:41:49.155858621 +0000 UTC m=+535.632141087" Aug 05 17:42:00.144086 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:42:00.144059 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v" Aug 05 17:42:53.949118 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:42:53.949089 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:42:53.950118 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:42:53.950093 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:43:01.506819 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:01.506776 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:43:18.427587 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:18.427498 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:43:27.100162 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:27.100124 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:43:45.734781 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.734746 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr"] Aug 05 17:43:45.738253 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.738219 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.741613 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.741592 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 05 17:43:45.742295 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.742279 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 05 17:43:45.742556 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.742541 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 05 17:43:45.742632 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.742591 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-274cz\"" Aug 05 17:43:45.748039 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.748019 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr"] Aug 05 17:43:45.769276 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.769397 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tn6\" (UniqueName: \"kubernetes.io/projected/8e0007cc-fca2-40bd-bfad-6c76833b8582-kube-api-access-h5tn6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.769397 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.769397 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0007cc-fca2-40bd-bfad-6c76833b8582-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.769499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769409 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.769499 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.769434 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870702 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870662 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tn6\" (UniqueName: \"kubernetes.io/projected/8e0007cc-fca2-40bd-bfad-6c76833b8582-kube-api-access-h5tn6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870702 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870968 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870723 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0007cc-fca2-40bd-bfad-6c76833b8582-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870968 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870968 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.870968 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.870801 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.871137 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.871121 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.871196 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.871174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.871257 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.871210 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.873259 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.873222 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0007cc-fca2-40bd-bfad-6c76833b8582-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.873429 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.873412 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0007cc-fca2-40bd-bfad-6c76833b8582-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:45.881617 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:45.881592 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tn6\" (UniqueName: \"kubernetes.io/projected/8e0007cc-fca2-40bd-bfad-6c76833b8582-kube-api-access-h5tn6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-c8kdr\" (UID: \"8e0007cc-fca2-40bd-bfad-6c76833b8582\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:46.048604 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:46.048560 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:46.183498 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:46.183468 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr"] Aug 05 17:43:46.185971 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:43:46.185941 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0007cc_fca2_40bd_bfad_6c76833b8582.slice/crio-d88721c8bccee6dd980c381c9b240797757fa660c617ebbc73c5ee5d09258e30 WatchSource:0}: Error finding container d88721c8bccee6dd980c381c9b240797757fa660c617ebbc73c5ee5d09258e30: Status 404 returned error can't find the container with id d88721c8bccee6dd980c381c9b240797757fa660c617ebbc73c5ee5d09258e30 Aug 05 17:43:46.306900 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:46.306813 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:43:46.562030 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:46.561944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" event={"ID":"8e0007cc-fca2-40bd-bfad-6c76833b8582","Type":"ContainerStarted","Data":"d88721c8bccee6dd980c381c9b240797757fa660c617ebbc73c5ee5d09258e30"} Aug 05 17:43:47.567522 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:47.567474 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" event={"ID":"8e0007cc-fca2-40bd-bfad-6c76833b8582","Type":"ContainerStarted","Data":"b0730f8341b6568d68428a43072d03267e58f7b7178252d14f3840187bbee4b6"} Aug 05 17:43:47.585802 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:47.585752 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" podStartSLOduration=1.6292047250000001 podStartE2EDuration="2.585738028s" totalImagesPullingTime="956.533303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:43:47.584571011 +0000 UTC m=+654.060853476" watchObservedRunningTime="2026-08-05 17:43:47.585738028 +0000 UTC m=+654.062020484" Aug 05 17:43:56.049528 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:56.049480 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:56.050137 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:56.049585 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:56.061946 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:56.061923 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:56.614823 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:56.614794 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-c8kdr" Aug 05 17:43:57.294674 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:43:57.294633 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:44:11.010848 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:44:11.010812 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:44:18.103946 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:44:18.103918 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:45:00.146727 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.146694 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd"] Aug 05 17:45:00.149736 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.149719 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" Aug 05 17:45:00.152463 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.152441 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 17:45:00.158546 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.158528 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" Aug 05 17:45:00.161940 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.161915 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd"] Aug 05 17:45:00.294273 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.294248 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd"] Aug 05 17:45:00.296658 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:45:00.296630 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c91b06_c868_4169_8a1d_b609937aee57.slice/crio-d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584 WatchSource:0}: Error finding container d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584: Status 404 returned error can't find the container with id d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584 Aug 05 17:45:00.298614 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.298597 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 17:45:00.854518 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:00.854472 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" event={"ID":"97c91b06-c868-4169-8a1d-b609937aee57","Type":"ContainerStarted","Data":"d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584"} Aug 05 17:45:03.868385 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:03.868294 2561 generic.go:356] "Generic (PLEG): container finished" podID="97c91b06-c868-4169-8a1d-b609937aee57" containerID="d695ac5caaba11cbd18ecf932ab8c576e3c84b7e9f66d92b13cfa3a39a0a2f67" exitCode=0 Aug 05 17:45:03.868385 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:03.868335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" event={"ID":"97c91b06-c868-4169-8a1d-b609937aee57","Type":"ContainerDied","Data":"d695ac5caaba11cbd18ecf932ab8c576e3c84b7e9f66d92b13cfa3a39a0a2f67"} Aug 05 17:45:05.000611 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:05.000575 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" Aug 05 17:45:05.878903 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:05.878873 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" Aug 05 17:45:05.879155 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:05.878907 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd" event={"ID":"97c91b06-c868-4169-8a1d-b609937aee57","Type":"ContainerDied","Data":"d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584"} Aug 05 17:45:05.879155 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:45:05.878949 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d08cd0dd887ac660f27422a24d9533e370426844761c2314057d7ec2adb584" Aug 05 17:46:00.805293 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:46:00.805220 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:46:10.500677 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:46:10.500643 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:46:24.096337 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:46:24.096256 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:47:53.972170 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:47:53.972077 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:47:53.972666 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:47:53.972342 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:48:04.101501 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:48:04.101468 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:48:14.900743 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:48:14.900709 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:49:19.317606 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:49:19.317518 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:49:25.603153 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:49:25.603117 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:49:41.201375 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:49:41.201333 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:49:48.106672 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:49:48.106630 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:49:57.607657 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:49:57.607625 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:07.299144 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:07.299100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:19.302750 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:19.302706 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:24.814672 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:24.814630 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:33.020818 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:33.020780 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:41.520046 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:41.520015 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:50:54.508374 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:50:54.508337 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:51:05.320262 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:51:05.320211 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:51:29.914095 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:51:29.914063 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:51:40.210959 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:51:40.210917 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:51:47.166996 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:51:47.166933 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:51:56.233345 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:51:56.233310 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:06.040358 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:06.040323 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:11.426772 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:11.426724 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:19.108819 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:19.108736 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:31.317180 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:31.317141 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:39.420262 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:39.420212 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:49.918788 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:49.918748 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:52:53.994714 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:53.994682 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:52:53.995424 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:53.995405 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:52:55.325825 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:52:55.325794 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:53:03.911813 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:53:03.911775 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:53:09.724074 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:53:09.724039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:53:14.920327 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:53:14.920288 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:53:25.652222 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:53:25.652188 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:55:42.802612 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:55:42.802525 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:55:48.126079 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:55:48.126040 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:56:14.314806 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:56:14.314768 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:56:43.738138 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:56:43.738099 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:56:50.411811 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:56:50.411726 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:56:59.720883 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:56:59.720848 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:05.012732 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:05.012689 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:16.330366 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:16.330329 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:25.498623 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:25.498583 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:31.241102 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:31.241066 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:36.343883 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:36.343844 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:45.909985 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:45.909948 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:52.903237 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:52.903199 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:57:54.015834 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:54.015797 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:57:54.023019 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:57:54.022989 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 17:58:03.014482 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:03.014443 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:58:28.602875 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:28.602783 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:58:33.900610 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:33.900571 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:58:39.678302 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.678267 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:58:39.678676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.678543 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c91b06-c868-4169-8a1d-b609937aee57" containerName="cleanup" Aug 05 17:58:39.678676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.678552 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c91b06-c868-4169-8a1d-b609937aee57" containerName="cleanup" Aug 05 17:58:39.678676 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.678601 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c91b06-c868-4169-8a1d-b609937aee57" containerName="cleanup" Aug 05 17:58:39.681496 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.681475 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.684559 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.684542 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7dd08526-kserve-self-signed-certs\"" Aug 05 17:58:39.710678 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.710647 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:58:39.784399 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.784399 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.784596 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.784596 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784479 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwps6\" (UniqueName: \"kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.784596 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784556 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.784596 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.784592 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.885884 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.885846 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.885884 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.885882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886093 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.885921 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886093 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.885947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwps6\" (UniqueName: \"kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886093 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.886024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886093 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.886046 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886378 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.886342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886428 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.886386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.886494 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.886478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.888410 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.888389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.888563 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.888545 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.896809 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.896783 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwps6\" (UniqueName: \"kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6\") pod \"e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:39.990589 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:39.990493 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:40.125049 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:40.125023 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:58:40.127444 ip-10-0-143-87 kubenswrapper[2561]: W0805 17:58:40.127412 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d7de720_a9f7_4462_a8e9_544946692462.slice/crio-4ad232e76e9c55748834f4c5cec888ca7d22f4ab2f54d144f3e705b46c2d13af WatchSource:0}: Error finding container 4ad232e76e9c55748834f4c5cec888ca7d22f4ab2f54d144f3e705b46c2d13af: Status 404 returned error can't find the container with id 4ad232e76e9c55748834f4c5cec888ca7d22f4ab2f54d144f3e705b46c2d13af Aug 05 17:58:41.086307 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:41.086268 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" event={"ID":"2d7de720-a9f7-4462-a8e9-544946692462","Type":"ContainerStarted","Data":"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b"} Aug 05 17:58:41.086307 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:41.086317 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" event={"ID":"2d7de720-a9f7-4462-a8e9-544946692462","Type":"ContainerStarted","Data":"4ad232e76e9c55748834f4c5cec888ca7d22f4ab2f54d144f3e705b46c2d13af"} Aug 05 17:58:41.105839 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:41.105777 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" podStartSLOduration=2.105762514 podStartE2EDuration="2.105762514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:58:41.104837638 +0000 UTC m=+1547.581120103" watchObservedRunningTime="2026-08-05 17:58:41.105762514 +0000 UTC m=+1547.582044979" Aug 05 17:58:41.430321 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:41.430216 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:58:49.991686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:49.991593 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:49.991686 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:49.991658 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:50.004189 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:50.004147 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:50.132940 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:50.132912 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:58:51.924879 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:58:51.924844 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:59:27.828670 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:27.828632 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:59:30.863745 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:30.863704 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:59:40.630011 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:40.629971 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:59:42.847464 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:42.847399 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" podUID="2d7de720-a9f7-4462-a8e9-544946692462" containerName="main" containerID="cri-o://dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b" gracePeriod=45 Aug 05 17:59:43.088381 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.088356 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:59:43.105224 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105176 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105224 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105242 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105268 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105307 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105405 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105352 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105551 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105409 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwps6\" (UniqueName: \"kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6\") pod \"2d7de720-a9f7-4462-a8e9-544946692462\" (UID: \"2d7de720-a9f7-4462-a8e9-544946692462\") " Aug 05 17:59:43.105613 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105542 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:59:43.105613 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105552 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home" (OuterVolumeSpecName: "home") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:59:43.105613 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105566 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:59:43.105757 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105658 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.105757 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105676 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.105757 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.105690 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.108328 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.108304 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm" (OuterVolumeSpecName: "dshm") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 17:59:43.108428 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.108364 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 17:59:43.108670 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.108637 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6" (OuterVolumeSpecName: "kube-api-access-vwps6") pod "2d7de720-a9f7-4462-a8e9-544946692462" (UID: "2d7de720-a9f7-4462-a8e9-544946692462"). InnerVolumeSpecName "kube-api-access-vwps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 17:59:43.206729 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.206679 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwps6\" (UniqueName: \"kubernetes.io/projected/2d7de720-a9f7-4462-a8e9-544946692462-kube-api-access-vwps6\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.206729 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.206725 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d7de720-a9f7-4462-a8e9-544946692462-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.206729 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.206740 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d7de720-a9f7-4462-a8e9-544946692462-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 17:59:43.336715 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.336685 2561 generic.go:356] "Generic (PLEG): container finished" podID="2d7de720-a9f7-4462-a8e9-544946692462" containerID="dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b" exitCode=0 Aug 05 17:59:43.336870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.336745 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" Aug 05 17:59:43.336870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.336771 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" event={"ID":"2d7de720-a9f7-4462-a8e9-544946692462","Type":"ContainerDied","Data":"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b"} Aug 05 17:59:43.336870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.336814 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6" event={"ID":"2d7de720-a9f7-4462-a8e9-544946692462","Type":"ContainerDied","Data":"4ad232e76e9c55748834f4c5cec888ca7d22f4ab2f54d144f3e705b46c2d13af"} Aug 05 17:59:43.336870 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.336831 2561 scope.go:117] "RemoveContainer" containerID="dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b" Aug 05 17:59:43.345134 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.345113 2561 scope.go:117] "RemoveContainer" containerID="dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b" Aug 05 17:59:43.345484 ip-10-0-143-87 kubenswrapper[2561]: E0805 17:59:43.345451 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b\": container with ID starting with dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b not found: ID does not exist" containerID="dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b" Aug 05 17:59:43.345614 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.345496 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b"} err="failed to get container status \"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b\": rpc error: code = NotFound desc = could not find container \"dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b\": container with ID starting with dd6aac588d01bfe875aa1f84c45a39d7c6d3f4faf9afc5818fa3779e95f6316b not found: ID does not exist" Aug 05 17:59:43.359592 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.359530 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:59:43.364355 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:43.364334 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7dd08526-kserve-7dc7899fff-4wcz6"] Aug 05 17:59:44.068970 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:44.068938 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7de720-a9f7-4462-a8e9-544946692462" path="/var/lib/kubelet/pods/2d7de720-a9f7-4462-a8e9-544946692462/volumes" Aug 05 17:59:46.823187 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:46.823105 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 17:59:57.411677 ip-10-0-143-87 kubenswrapper[2561]: I0805 17:59:57.411644 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:00.142092 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.142004 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs"] Aug 05 18:00:00.142640 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.142344 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7de720-a9f7-4462-a8e9-544946692462" containerName="main" Aug 05 18:00:00.142640 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.142359 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7de720-a9f7-4462-a8e9-544946692462" containerName="main" Aug 05 18:00:00.142640 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.142407 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7de720-a9f7-4462-a8e9-544946692462" containerName="main" Aug 05 18:00:00.146324 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.146302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" Aug 05 18:00:00.149033 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.149011 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 18:00:00.156256 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.156209 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs"] Aug 05 18:00:00.156447 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.156429 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" Aug 05 18:00:00.287013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.286986 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs"] Aug 05 18:00:00.291738 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:00:00.291707 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78af49b_0cda_4735_8755_0a3282baaff6.slice/crio-e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df WatchSource:0}: Error finding container e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df: Status 404 returned error can't find the container with id e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df Aug 05 18:00:00.409778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:00.409692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" event={"ID":"a78af49b-0cda-4735-8755-0a3282baaff6","Type":"ContainerStarted","Data":"e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df"} Aug 05 18:00:01.416290 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:01.416197 2561 generic.go:356] "Generic (PLEG): container finished" podID="a78af49b-0cda-4735-8755-0a3282baaff6" containerID="ecf336bb830aed34345f8592e85337d21f05a5322248d4da382e8165bd092eca" exitCode=0 Aug 05 18:00:01.416636 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:01.416283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" event={"ID":"a78af49b-0cda-4735-8755-0a3282baaff6","Type":"ContainerDied","Data":"ecf336bb830aed34345f8592e85337d21f05a5322248d4da382e8165bd092eca"} Aug 05 18:00:02.552582 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:02.552559 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" Aug 05 18:00:03.425360 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:03.425320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" event={"ID":"a78af49b-0cda-4735-8755-0a3282baaff6","Type":"ContainerDied","Data":"e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df"} Aug 05 18:00:03.425537 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:03.425379 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e813cd23fac90a2a33846c8dd9798cf77f6096f01c4bfd02b602abc5a8a124df" Aug 05 18:00:03.425537 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:03.425382 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765880-5c6zs" Aug 05 18:00:04.406518 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:04.406483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:11.411203 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:11.411153 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:22.900453 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:22.900415 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:29.712522 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:29.712482 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:40.010416 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:40.010379 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:48.608184 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:48.608144 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:00:57.525582 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:00:57.525546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:03.103769 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:03.103735 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:10.024369 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:10.024332 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:21.219213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:21.219132 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:28.427270 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:28.427217 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:35.307913 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:35.307877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:39.326151 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:39.326115 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:47.054682 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:47.054644 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:01:55.905381 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:01:55.905345 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:03.661337 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:03.661302 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:13.368533 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:13.368498 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:24.249646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:24.249608 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:31.023560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:31.023524 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:45.736212 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:45.736128 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:52.003300 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:52.003266 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:02:54.043385 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:54.043360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:02:54.045338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:54.045314 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:02:59.016467 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:02:59.016430 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:03:09.921661 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:09.921623 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:03:18.542118 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:18.542080 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:03:29.831481 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:29.831438 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:03:33.487403 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.487353 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:03:33.487977 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.487754 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78af49b-0cda-4735-8755-0a3282baaff6" containerName="cleanup" Aug 05 18:03:33.487977 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.487772 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78af49b-0cda-4735-8755-0a3282baaff6" containerName="cleanup" Aug 05 18:03:33.487977 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.487863 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78af49b-0cda-4735-8755-0a3282baaff6" containerName="cleanup" Aug 05 18:03:33.490883 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.490860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.497654 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.497630 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8effb523-openshift-default-dockercfg-c5tdg\"" Aug 05 18:03:33.553495 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553457 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553495 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553493 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpdx\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553606 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553656 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553858 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.553858 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.553727 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.654806 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654773 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.654806 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654861 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.654982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpdx\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655439 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.655289 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655439 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.655345 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655537 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.655466 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655630 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.655599 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.655755 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.655687 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.657754 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.657735 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.658097 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.658076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.672579 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.672554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:03:33.703055 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.703024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.728325 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.728298 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpdx\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx\") pod \"e2e-ait-8effb523-openshift-default-685f47775d-hzrdt\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:33.800824 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:33.800783 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:34.005703 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:34.005672 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:03:34.006595 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:03:34.006570 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c971eb_87db_44ba_b97a_a45ff8219858.slice/crio-bd3c29fde10cf076ac83776d20618825dacf80792705661d0996e1633bf38e54 WatchSource:0}: Error finding container bd3c29fde10cf076ac83776d20618825dacf80792705661d0996e1633bf38e54: Status 404 returned error can't find the container with id bd3c29fde10cf076ac83776d20618825dacf80792705661d0996e1633bf38e54 Aug 05 18:03:34.008589 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:34.008572 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 05 18:03:34.275715 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:34.275661 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerStarted","Data":"bd3c29fde10cf076ac83776d20618825dacf80792705661d0996e1633bf38e54"} Aug 05 18:03:37.731533 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:37.731488 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:37.731865 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:37.731567 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:37.731865 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:37.731598 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:38.295507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:38.295463 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerStarted","Data":"7dbf4e587624c663df76ca3c197f4d227962767f42ed7a497aa01d688fe3d71f"} Aug 05 18:03:38.340933 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:38.340880 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podStartSLOduration=1.6183990129999999 podStartE2EDuration="5.340865417s" totalImagesPullingTime="3.722466404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:03:38.340702716 +0000 UTC m=+1844.816985181" watchObservedRunningTime="2026-08-05 18:03:38.340865417 +0000 UTC m=+1844.817147881" Aug 05 18:03:38.800941 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:38.800902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:38.806217 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:38.806189 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:39.300097 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:39.300066 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:39.301155 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:39.301135 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:45.548612 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:45.548572 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:03:55.185577 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.185532 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": read tcp 10.132.0.2:59226->10.132.0.25:15021: read: connection reset by peer" start-of-body= Aug 05 18:03:55.186010 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.185619 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": read tcp 10.132.0.2:59226->10.132.0.25:15021: read: connection reset by peer" Aug 05 18:03:55.368343 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.368253 2561 generic.go:356] "Generic (PLEG): container finished" podID="48c971eb-87db-44ba-b97a-a45ff8219858" containerID="7dbf4e587624c663df76ca3c197f4d227962767f42ed7a497aa01d688fe3d71f" exitCode=0 Aug 05 18:03:55.368480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.368259 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerDied","Data":"7dbf4e587624c663df76ca3c197f4d227962767f42ed7a497aa01d688fe3d71f"} Aug 05 18:03:55.368761 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.368747 2561 scope.go:117] "RemoveContainer" containerID="7dbf4e587624c663df76ca3c197f4d227962767f42ed7a497aa01d688fe3d71f" Aug 05 18:03:55.369761 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.369725 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:55.369822 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.369804 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:55.369860 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:55.369839 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:03:56.374408 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:56.374372 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerStarted","Data":"6100870e14e6f55817c81b31327e7ceaeb74de2484ff4da4c00d748d570d9099"} Aug 05 18:03:56.801545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:56.801506 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:03:56.801910 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:56.801885 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Aug 05 18:03:56.802095 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:56.802067 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Aug 05 18:03:57.801270 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:57.801205 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Aug 05 18:03:57.801860 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:57.801826 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Aug 05 18:03:58.802202 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:58.802160 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Aug 05 18:03:58.802835 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:58.802799 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Aug 05 18:03:59.801914 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:59.801861 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Aug 05 18:03:59.802123 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:03:59.801949 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Aug 05 18:04:00.801695 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:00.801642 2561 patch_prober.go:28] interesting pod/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body= Aug 05 18:04:00.802248 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:00.801734 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" Aug 05 18:04:01.805144 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:01.805114 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:04:01.805543 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:01.805468 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:04:01.806162 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:01.806135 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:04:22.361996 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:22.361911 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:04:22.362461 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:22.362119 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" containerID="cri-o://6100870e14e6f55817c81b31327e7ceaeb74de2484ff4da4c00d748d570d9099" gracePeriod=30 Aug 05 18:04:23.483573 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.483539 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:04:23.488424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.488400 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.492295 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.492270 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e7e381dc-gw-openshift-default-dockercfg-pgsm4\"" Aug 05 18:04:23.503606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.503574 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:04:23.602305 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602266 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7p2\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602305 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602328 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602419 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602485 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.602531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.602501 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7p2\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703649 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703716 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703763 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.703889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.703821 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.704263 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.704165 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.704263 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.704210 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.704351 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.704277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.704415 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.704395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.704460 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.704441 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.706394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.706362 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.706595 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.706576 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.713920 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.713890 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.714608 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.714586 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7p2\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2\") pod \"e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.798363 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.798314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:23.945266 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.945211 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:04:23.948708 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:04:23.948675 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2a2bc1_831f_430d_929f_e6c02658b5f5.slice/crio-fe018d39f5afb576e205fcb900e494a3592586884c3513c87027d1ae342c4ab6 WatchSource:0}: Error finding container fe018d39f5afb576e205fcb900e494a3592586884c3513c87027d1ae342c4ab6: Status 404 returned error can't find the container with id fe018d39f5afb576e205fcb900e494a3592586884c3513c87027d1ae342c4ab6 Aug 05 18:04:23.950819 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.950785 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:23.950915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.950871 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:23.950974 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:23.950917 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:24.492254 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:24.492196 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerStarted","Data":"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e"} Aug 05 18:04:24.492636 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:24.492272 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerStarted","Data":"fe018d39f5afb576e205fcb900e494a3592586884c3513c87027d1ae342c4ab6"} Aug 05 18:04:24.516957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:24.516907 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podStartSLOduration=1.516891738 podStartE2EDuration="1.516891738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:04:24.516597522 +0000 UTC m=+1890.992879997" watchObservedRunningTime="2026-08-05 18:04:24.516891738 +0000 UTC m=+1890.993174202" Aug 05 18:04:24.798735 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:24.798694 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:24.803799 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:24.803772 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:25.106176 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:25.106089 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:04:25.497173 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:25.497079 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:25.498161 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:25.498138 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:27.507487 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.507454 2561 generic.go:356] "Generic (PLEG): container finished" podID="48c971eb-87db-44ba-b97a-a45ff8219858" containerID="6100870e14e6f55817c81b31327e7ceaeb74de2484ff4da4c00d748d570d9099" exitCode=0 Aug 05 18:04:27.507943 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.507517 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerDied","Data":"6100870e14e6f55817c81b31327e7ceaeb74de2484ff4da4c00d748d570d9099"} Aug 05 18:04:27.507943 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.507621 2561 scope.go:117] "RemoveContainer" containerID="7dbf4e587624c663df76ca3c197f4d227962767f42ed7a497aa01d688fe3d71f" Aug 05 18:04:27.608202 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.608178 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:04:27.737177 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737094 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737177 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737138 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737177 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737170 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdpdx\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737243 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737261 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737284 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737326 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737346 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737370 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs\") pod \"48c971eb-87db-44ba-b97a-a45ff8219858\" (UID: \"48c971eb-87db-44ba-b97a-a45ff8219858\") " Aug 05 18:04:27.737477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737443 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:04:27.737800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737599 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.737800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737675 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data" (OuterVolumeSpecName: "istio-data") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:04:27.737800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737712 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:04:27.737800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737733 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:04:27.737800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.737781 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:04:27.739759 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.739730 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:04:27.739895 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.739865 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token" (OuterVolumeSpecName: "istio-token") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:04:27.740044 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.740024 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:04:27.740097 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.740081 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx" (OuterVolumeSpecName: "kube-api-access-xdpdx") pod "48c971eb-87db-44ba-b97a-a45ff8219858" (UID: "48c971eb-87db-44ba-b97a-a45ff8219858"). InnerVolumeSpecName "kube-api-access-xdpdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:04:27.838376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838341 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c971eb-87db-44ba-b97a-a45ff8219858-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838368 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838394 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838402 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838411 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpdx\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-kube-api-access-xdpdx\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838419 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c971eb-87db-44ba-b97a-a45ff8219858-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838427 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c971eb-87db-44ba-b97a-a45ff8219858-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:27.838570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:27.838434 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c971eb-87db-44ba-b97a-a45ff8219858-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:04:28.513426 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:28.513399 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" Aug 05 18:04:28.513929 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:28.513427 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt" event={"ID":"48c971eb-87db-44ba-b97a-a45ff8219858","Type":"ContainerDied","Data":"bd3c29fde10cf076ac83776d20618825dacf80792705661d0996e1633bf38e54"} Aug 05 18:04:28.513929 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:28.513492 2561 scope.go:117] "RemoveContainer" containerID="6100870e14e6f55817c81b31327e7ceaeb74de2484ff4da4c00d748d570d9099" Aug 05 18:04:28.531019 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:28.530995 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:04:28.534879 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:28.534851 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8effb523-openshift-default-685f47775d-hzrdt"] Aug 05 18:04:30.069909 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:30.069869 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" path="/var/lib/kubelet/pods/48c971eb-87db-44ba-b97a-a45ff8219858/volumes" Aug 05 18:04:32.114366 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:32.114317 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:04:39.564050 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.564019 2561 generic.go:356] "Generic (PLEG): container finished" podID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerID="dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e" exitCode=0 Aug 05 18:04:39.564525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.564067 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerDied","Data":"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e"} Aug 05 18:04:39.564525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.564433 2561 scope.go:117] "RemoveContainer" containerID="dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e" Aug 05 18:04:39.565492 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.565466 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:39.565560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.565532 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:39.565593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:39.565562 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:40.497743 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:40.497626 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:40.570176 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:40.570129 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerStarted","Data":"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f"} Aug 05 18:04:40.800193 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:40.800080 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:40.800401 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:40.800260 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Aug 05 18:04:40.800401 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:40.800325 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Aug 05 18:04:41.799873 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:41.799830 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Aug 05 18:04:41.800532 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:41.800496 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Aug 05 18:04:42.799786 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:42.799743 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Aug 05 18:04:42.800083 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:42.800050 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Aug 05 18:04:43.799785 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:43.799743 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Aug 05 18:04:43.800093 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:43.800058 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Aug 05 18:04:44.798827 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:44.798781 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body= Aug 05 18:04:44.799159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:44.799129 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" Aug 05 18:04:45.803384 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:45.803350 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:45.803760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:45.803698 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:45.804541 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:45.804517 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:04:58.109058 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.109007 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:04:58.109654 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.109299 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" containerID="cri-o://cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f" gracePeriod=30 Aug 05 18:04:58.841854 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.841807 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:04:58.842124 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842112 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.842250 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842126 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.842250 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842137 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.842250 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842157 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.842250 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842208 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.842250 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.842215 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c971eb-87db-44ba-b97a-a45ff8219858" containerName="istio-proxy" Aug 05 18:04:58.843971 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.843953 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.846829 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.846809 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-efc0f562-openshift-default-dockercfg-5nsgw\"" Aug 05 18:04:58.866736 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.866699 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:04:58.901342 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901342 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901340 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901364 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901391 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7twg\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901560 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901505 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901712 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901623 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:58.901712 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:58.901650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002357 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002313 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002357 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002426 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002515 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002550 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002582 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7twg\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002795 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.002982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.002911 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.003156 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.003133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.003202 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.003152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.003378 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.003360 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.005083 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.005063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.005148 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.005090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.011966 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.011941 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.011966 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.011960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7twg\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg\") pod \"e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.155119 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.155003 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:04:59.294131 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.294106 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:04:59.297863 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:04:59.297827 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e673430_6985_4c6d_b610_6ed1ef74b93c.slice/crio-d58e1b493410f560d47c1e0d434a802efb27ad7b36dc4ecc160f79cd99507baa WatchSource:0}: Error finding container d58e1b493410f560d47c1e0d434a802efb27ad7b36dc4ecc160f79cd99507baa: Status 404 returned error can't find the container with id d58e1b493410f560d47c1e0d434a802efb27ad7b36dc4ecc160f79cd99507baa Aug 05 18:04:59.300565 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.300525 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:59.300663 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.300628 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:59.300705 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.300661 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:04:59.655061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.655023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerStarted","Data":"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57"} Aug 05 18:04:59.655296 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.655084 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerStarted","Data":"d58e1b493410f560d47c1e0d434a802efb27ad7b36dc4ecc160f79cd99507baa"} Aug 05 18:04:59.685249 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:04:59.685178 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podStartSLOduration=1.685160518 podStartE2EDuration="1.685160518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:04:59.684194679 +0000 UTC m=+1926.160477145" watchObservedRunningTime="2026-08-05 18:04:59.685160518 +0000 UTC m=+1926.161442962" Aug 05 18:05:00.156020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.155987 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:00.160833 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.160807 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:00.665432 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.665404 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:00.666540 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.666513 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:00.806866 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.806820 2561 patch_prober.go:28] interesting pod/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:05:00.807030 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:00.806884 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:05:01.062608 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:01.062573 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:05:03.368982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.368955 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:05:03.442286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442161 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442213 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442269 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd7p2\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442370 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442437 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442469 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442502 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442541 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442830 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442573 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs\") pod \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\" (UID: \"7b2a2bc1-831f-430d-929f-e6c02658b5f5\") " Aug 05 18:05:03.442830 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442578 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:03.442830 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442675 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:05:03.442830 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442745 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:03.443037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442851 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:03.443037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442864 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.443037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442887 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.443037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.442901 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.443037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.443022 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:03.444765 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.444732 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:03.444987 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.444958 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2" (OuterVolumeSpecName: "kube-api-access-zd7p2") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "kube-api-access-zd7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:03.445262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.445212 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:03.445352 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.445219 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b2a2bc1-831f-430d-929f-e6c02658b5f5" (UID: "7b2a2bc1-831f-430d-929f-e6c02658b5f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:05:03.543694 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543653 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.543694 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543681 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.543694 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543692 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.543694 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543701 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7p2\" (UniqueName: \"kubernetes.io/projected/7b2a2bc1-831f-430d-929f-e6c02658b5f5-kube-api-access-zd7p2\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.543954 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543711 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.543954 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.543719 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2a2bc1-831f-430d-929f-e6c02658b5f5-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:03.680538 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.680503 2561 generic.go:356] "Generic (PLEG): container finished" podID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerID="cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f" exitCode=0 Aug 05 18:05:03.680715 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.680551 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" Aug 05 18:05:03.680715 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.680560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerDied","Data":"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f"} Aug 05 18:05:03.680715 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.680605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l" event={"ID":"7b2a2bc1-831f-430d-929f-e6c02658b5f5","Type":"ContainerDied","Data":"fe018d39f5afb576e205fcb900e494a3592586884c3513c87027d1ae342c4ab6"} Aug 05 18:05:03.680715 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.680637 2561 scope.go:117] "RemoveContainer" containerID="cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f" Aug 05 18:05:03.689726 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.689707 2561 scope.go:117] "RemoveContainer" containerID="dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e" Aug 05 18:05:03.697558 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.697533 2561 scope.go:117] "RemoveContainer" containerID="cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f" Aug 05 18:05:03.697806 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:05:03.697788 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f\": container with ID starting with cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f not found: ID does not exist" containerID="cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f" Aug 05 18:05:03.697851 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.697814 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f"} err="failed to get container status \"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f\": rpc error: code = NotFound desc = could not find container \"cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f\": container with ID starting with cc149c52d78895278f8d8d46d149b5a0e7874b8f93ad23255e81964d57358b3f not found: ID does not exist" Aug 05 18:05:03.697851 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.697835 2561 scope.go:117] "RemoveContainer" containerID="dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e" Aug 05 18:05:03.698061 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:05:03.698042 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e\": container with ID starting with dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e not found: ID does not exist" containerID="dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e" Aug 05 18:05:03.698101 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.698067 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e"} err="failed to get container status \"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e\": rpc error: code = NotFound desc = could not find container \"dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e\": container with ID starting with dbcec457f69093d2050303e05db5d735d99c63f6862a0bbfdd5223d675429b7e not found: ID does not exist" Aug 05 18:05:03.709678 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.709653 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:05:03.712104 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:03.712079 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e7e381dc-gw-openshift-default-556845855b-bd46l"] Aug 05 18:05:04.068297 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:04.068259 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" path="/var/lib/kubelet/pods/7b2a2bc1-831f-430d-929f-e6c02658b5f5/volumes" Aug 05 18:05:10.107155 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:10.107108 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:05:15.667048 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.667009 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:15.667501 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.667078 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:15.742172 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.742135 2561 generic.go:356] "Generic (PLEG): container finished" podID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerID="54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57" exitCode=0 Aug 05 18:05:15.742353 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.742193 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerDied","Data":"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57"} Aug 05 18:05:15.742605 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.742589 2561 scope.go:117] "RemoveContainer" containerID="54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57" Aug 05 18:05:15.743658 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.743605 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:15.743731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.743715 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:15.743777 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:15.743755 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:16.748307 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:16.748273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerStarted","Data":"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348"} Aug 05 18:05:17.156262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:17.156154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:17.156445 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:17.156425 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:17.156497 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:17.156477 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:18.155446 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:18.155407 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:18.155816 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:18.155469 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:19.155559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:19.155516 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:19.155945 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:19.155578 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:20.155642 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:20.155607 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:20.156051 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:20.155662 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:21.155672 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:21.155632 2561 patch_prober.go:28] interesting pod/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body= Aug 05 18:05:21.156087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:21.155695 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" Aug 05 18:05:22.160115 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:22.160084 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:22.160538 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:22.160437 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:22.161185 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:22.161166 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:44.264702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:44.264668 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:05:44.265195 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:44.264938 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" containerID="cri-o://e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348" gracePeriod=30 Aug 05 18:05:45.012854 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.012816 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:05:45.013105 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013093 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.013147 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013107 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.013147 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013126 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.013147 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013131 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.013266 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013180 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.013266 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.013187 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2a2bc1-831f-430d-929f-e6c02658b5f5" containerName="istio-proxy" Aug 05 18:05:45.015020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.015003 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.017525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.017502 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-65feef49-openshift-default-dockercfg-bbk7s\"" Aug 05 18:05:45.037647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.037624 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:05:45.100734 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100701 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.100925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fc6\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.100925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.100925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.100925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.100925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.101139 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100947 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.101139 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.100997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.101139 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.101074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202194 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202209 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202255 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fc6\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202330 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202624 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202396 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202624 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202624 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202624 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202752 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.202858 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.203106 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202866 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.203106 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.202966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.204791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.204766 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.205386 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.205368 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.222527 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.222499 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.222789 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.222767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fc6\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6\") pod \"e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.324851 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.324765 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:45.475742 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.475718 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:05:45.479297 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:05:45.479255 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e801e9_e951_44ae_a140_3b6d3e8702bc.slice/crio-dac8dfc51a1fffb6bbace69717b9f6a1870623b252562ea512a5c131288bbdcc WatchSource:0}: Error finding container dac8dfc51a1fffb6bbace69717b9f6a1870623b252562ea512a5c131288bbdcc: Status 404 returned error can't find the container with id dac8dfc51a1fffb6bbace69717b9f6a1870623b252562ea512a5c131288bbdcc Aug 05 18:05:45.481801 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.481768 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:45.481893 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.481838 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:45.481893 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.481868 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:05:45.872512 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.872477 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerStarted","Data":"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21"} Aug 05 18:05:45.872700 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.872527 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerStarted","Data":"dac8dfc51a1fffb6bbace69717b9f6a1870623b252562ea512a5c131288bbdcc"} Aug 05 18:05:45.901084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:45.901027 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podStartSLOduration=1.9010112750000001 podStartE2EDuration="1.901011275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:05:45.899307137 +0000 UTC m=+1972.375589620" watchObservedRunningTime="2026-08-05 18:05:45.901011275 +0000 UTC m=+1972.377293813" Aug 05 18:05:46.004923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:46.004889 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:05:46.325792 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:46.325755 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:46.330644 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:46.330620 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:46.877060 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:46.877030 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:46.878096 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:46.878079 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:05:49.508962 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.508937 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:49.642374 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642286 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642374 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642323 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642374 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642361 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642391 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642425 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7twg\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642456 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642508 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642578 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642637 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642622 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert\") pod \"3e673430-6985-4c6d-b610-6ed1ef74b93c\" (UID: \"3e673430-6985-4c6d-b610-6ed1ef74b93c\") " Aug 05 18:05:49.642924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642667 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:49.642924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642676 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:49.642924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642773 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:49.642924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642870 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:49.643087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642968 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.643087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.642986 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.643087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.643001 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.643087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.643014 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.643087 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.643048 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:05:49.644954 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.644923 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:05:49.645061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.645024 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg" (OuterVolumeSpecName: "kube-api-access-t7twg") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "kube-api-access-t7twg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:49.645344 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.645320 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:05:49.645344 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.645334 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e673430-6985-4c6d-b610-6ed1ef74b93c" (UID: "3e673430-6985-4c6d-b610-6ed1ef74b93c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:05:49.743606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.743566 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.743606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.743600 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e673430-6985-4c6d-b610-6ed1ef74b93c-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.743606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.743609 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.743606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.743618 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7twg\" (UniqueName: \"kubernetes.io/projected/3e673430-6985-4c6d-b610-6ed1ef74b93c-kube-api-access-t7twg\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.743871 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.743627 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e673430-6985-4c6d-b610-6ed1ef74b93c-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:05:49.891801 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.891764 2561 generic.go:356] "Generic (PLEG): container finished" podID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerID="e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348" exitCode=0 Aug 05 18:05:49.891970 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.891815 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" Aug 05 18:05:49.891970 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.891811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerDied","Data":"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348"} Aug 05 18:05:49.891970 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.891874 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr" event={"ID":"3e673430-6985-4c6d-b610-6ed1ef74b93c","Type":"ContainerDied","Data":"d58e1b493410f560d47c1e0d434a802efb27ad7b36dc4ecc160f79cd99507baa"} Aug 05 18:05:49.891970 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.891900 2561 scope.go:117] "RemoveContainer" containerID="e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348" Aug 05 18:05:49.900880 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.900861 2561 scope.go:117] "RemoveContainer" containerID="54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57" Aug 05 18:05:49.908523 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.908504 2561 scope.go:117] "RemoveContainer" containerID="e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348" Aug 05 18:05:49.908804 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:05:49.908785 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348\": container with ID starting with e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348 not found: ID does not exist" containerID="e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348" Aug 05 18:05:49.908880 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.908817 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348"} err="failed to get container status \"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348\": rpc error: code = NotFound desc = could not find container \"e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348\": container with ID starting with e90e4563103b76fda8305effee62f8ea42b1d450930144d3dd349b5b9c9ce348 not found: ID does not exist" Aug 05 18:05:49.908880 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.908843 2561 scope.go:117] "RemoveContainer" containerID="54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57" Aug 05 18:05:49.909099 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:05:49.909080 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57\": container with ID starting with 54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57 not found: ID does not exist" containerID="54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57" Aug 05 18:05:49.909140 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.909105 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57"} err="failed to get container status \"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57\": rpc error: code = NotFound desc = could not find container \"54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57\": container with ID starting with 54e50244cf80d578ac06fb8ba2d6736d01de33e41d0c4cfd08b6b593390d0a57 not found: ID does not exist" Aug 05 18:05:49.921491 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.921468 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:05:49.925472 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:49.925447 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-efc0f562-openshift-default-6fbb54b994-sb8hr"] Aug 05 18:05:50.069619 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:50.069588 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" path="/var/lib/kubelet/pods/3e673430-6985-4c6d-b610-6ed1ef74b93c/volumes" Aug 05 18:05:55.320341 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:05:55.320307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:06:00.942363 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.942329 2561 generic.go:356] "Generic (PLEG): container finished" podID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerID="5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21" exitCode=0 Aug 05 18:06:00.942817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.942411 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerDied","Data":"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21"} Aug 05 18:06:00.942817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.942794 2561 scope.go:117] "RemoveContainer" containerID="5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21" Aug 05 18:06:00.943754 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.943715 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:00.943882 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.943802 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:00.943882 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:00.943850 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:06:01.877382 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:01.877332 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:01.948768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:01.948731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerStarted","Data":"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8"} Aug 05 18:06:02.325766 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:02.325727 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:02.326985 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:02.326959 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Aug 05 18:06:02.327089 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:02.327011 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Aug 05 18:06:03.325406 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:03.325366 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Aug 05 18:06:03.325898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:03.325452 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Aug 05 18:06:04.325726 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:04.325683 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Aug 05 18:06:04.326142 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:04.325747 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Aug 05 18:06:05.325173 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:05.325135 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Aug 05 18:06:05.325377 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:05.325191 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Aug 05 18:06:06.325245 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:06.325188 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Aug 05 18:06:06.325629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:06.325271 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Aug 05 18:06:07.329554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:07.329525 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:07.330009 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:07.329905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:07.330545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:07.330525 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:19.388635 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:19.388581 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:06:19.389295 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:19.388904 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" containerID="cri-o://28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8" gracePeriod=30 Aug 05 18:06:22.018652 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:22.018614 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:06:22.332942 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:22.332857 2561 patch_prober.go:28] interesting pod/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 05 18:06:22.332942 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:22.332917 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:06:24.635846 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.635819 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:24.753830 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.753805 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.753999 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.753844 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.753999 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.753873 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.753999 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.753888 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.753999 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.753902 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754051 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754120 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754156 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9fc6\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754164 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754180 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:24.754218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754188 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert\") pod \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\" (UID: \"d3e801e9-e951-44ae-a140-3b6d3e8702bc\") " Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754330 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754394 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754494 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754532 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754548 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.754559 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754561 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.755019 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.754573 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.756268 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.756245 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:06:24.756362 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.756284 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:06:24.756362 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.756356 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:06:24.756552 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.756534 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6" (OuterVolumeSpecName: "kube-api-access-n9fc6") pod "d3e801e9-e951-44ae-a140-3b6d3e8702bc" (UID: "d3e801e9-e951-44ae-a140-3b6d3e8702bc"). InnerVolumeSpecName "kube-api-access-n9fc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:06:24.855867 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.855827 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.855867 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.855857 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.855867 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.855866 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.855867 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.855875 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9fc6\" (UniqueName: \"kubernetes.io/projected/d3e801e9-e951-44ae-a140-3b6d3e8702bc-kube-api-access-n9fc6\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:24.856131 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:24.855884 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e801e9-e951-44ae-a140-3b6d3e8702bc-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:06:25.044450 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.044364 2561 generic.go:356] "Generic (PLEG): container finished" podID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerID="28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8" exitCode=0 Aug 05 18:06:25.044450 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.044441 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerDied","Data":"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8"} Aug 05 18:06:25.044673 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.044481 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" event={"ID":"d3e801e9-e951-44ae-a140-3b6d3e8702bc","Type":"ContainerDied","Data":"dac8dfc51a1fffb6bbace69717b9f6a1870623b252562ea512a5c131288bbdcc"} Aug 05 18:06:25.044673 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.044488 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv" Aug 05 18:06:25.044673 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.044503 2561 scope.go:117] "RemoveContainer" containerID="28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8" Aug 05 18:06:25.054984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.054964 2561 scope.go:117] "RemoveContainer" containerID="5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21" Aug 05 18:06:25.062674 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.062655 2561 scope.go:117] "RemoveContainer" containerID="28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8" Aug 05 18:06:25.062951 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:06:25.062933 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8\": container with ID starting with 28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8 not found: ID does not exist" containerID="28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8" Aug 05 18:06:25.063054 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.062961 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8"} err="failed to get container status \"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8\": rpc error: code = NotFound desc = could not find container \"28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8\": container with ID starting with 28cc57081e961a8f0f3795437b7dc272d9997331c6ac7826eab2cf7fb26a53a8 not found: ID does not exist" Aug 05 18:06:25.063054 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.062979 2561 scope.go:117] "RemoveContainer" containerID="5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21" Aug 05 18:06:25.063211 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:06:25.063194 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21\": container with ID starting with 5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21 not found: ID does not exist" containerID="5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21" Aug 05 18:06:25.063282 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.063217 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21"} err="failed to get container status \"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21\": rpc error: code = NotFound desc = could not find container \"5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21\": container with ID starting with 5461b082471660dfe6f997d439af6106e2275d9f91c69cb253039bd0cdf1cc21 not found: ID does not exist" Aug 05 18:06:25.075424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.075403 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:06:25.080467 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:25.080447 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-65feef49-openshift-default-5479d755b8-6cpjv"] Aug 05 18:06:26.069007 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:26.068963 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" path="/var/lib/kubelet/pods/d3e801e9-e951-44ae-a140-3b6d3e8702bc/volumes" Aug 05 18:06:34.999842 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:06:34.999803 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:07:54.064551 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:07:54.064520 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:07:54.068443 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:07:54.068418 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:08:06.820104 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:06.820068 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:08:15.204654 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:15.204608 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:08:22.421984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.421949 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422215 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422240 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422248 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422254 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422262 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422267 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422272 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422277 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422328 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:22.422368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.422338 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:22.424316 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.424300 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.426834 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.426811 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2e99d153-openshift-default-dockercfg-rrd5q\"" Aug 05 18:08:22.478650 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478616 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478650 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478649 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z886\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478813 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.478837 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.479036 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.478865 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.500387 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.500354 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:08:22.579423 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579535 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z886\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579568 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.579616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579604 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579641 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579844 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579939 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.579987 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580322 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.580098 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.580435 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.580414 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.581925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.581905 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.582129 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.582113 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.589048 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.589026 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.589208 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.589187 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z886\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886\") pod \"e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.734167 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.734081 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:22.872258 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.868905 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:08:22.875729 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:08:22.875696 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9768c2dd_3afc_4e32_83e3_376e6af1345d.slice/crio-a32fcf3373e4d25bdae7745441837a0be58856e08261f38c14cb8864bab31a89 WatchSource:0}: Error finding container a32fcf3373e4d25bdae7745441837a0be58856e08261f38c14cb8864bab31a89: Status 404 returned error can't find the container with id a32fcf3373e4d25bdae7745441837a0be58856e08261f38c14cb8864bab31a89 Aug 05 18:08:22.877960 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.877920 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:22.878025 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.877996 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:22.878062 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:22.878024 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:23.525762 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:23.525726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerStarted","Data":"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4"} Aug 05 18:08:23.526143 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:23.525779 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerStarted","Data":"a32fcf3373e4d25bdae7745441837a0be58856e08261f38c14cb8864bab31a89"} Aug 05 18:08:23.559269 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:23.559195 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podStartSLOduration=1.559176154 podStartE2EDuration="1.559176154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:08:23.558353782 +0000 UTC m=+2130.034636252" watchObservedRunningTime="2026-08-05 18:08:23.559176154 +0000 UTC m=+2130.035458621" Aug 05 18:08:23.734891 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:23.734854 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:23.740161 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:23.740132 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:24.529691 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:24.529658 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:24.530634 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:24.530614 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:30.001290 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:30.001255 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:08:39.531341 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.531306 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:39.531744 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.531374 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:39.589358 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.589273 2561 generic.go:356] "Generic (PLEG): container finished" podID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerID="1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4" exitCode=0 Aug 05 18:08:39.589358 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.589311 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerDied","Data":"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4"} Aug 05 18:08:39.589687 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.589673 2561 scope.go:117] "RemoveContainer" containerID="1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4" Aug 05 18:08:39.590665 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.590635 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:39.590724 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.590701 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:39.590759 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:39.590734 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:08:40.599029 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:40.598960 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerStarted","Data":"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7"} Aug 05 18:08:40.734745 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:40.734701 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:40.735006 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:40.734985 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:40.735055 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:40.735033 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:41.734875 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:41.734835 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:41.735289 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:41.734895 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:42.735322 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:42.735288 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:42.735756 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:42.735343 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:43.735211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:43.735166 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:43.735425 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:43.735257 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:44.735470 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:44.735427 2561 patch_prober.go:28] interesting pod/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Aug 05 18:08:44.735885 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:44.735486 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Aug 05 18:08:45.738800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:45.738766 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:45.739213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:45.739095 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:45.739943 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:45.739926 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:08:58.121484 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.121452 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t"] Aug 05 18:08:58.121915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.121779 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e673430-6985-4c6d-b610-6ed1ef74b93c" containerName="istio-proxy" Aug 05 18:08:58.121915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.121793 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e801e9-e951-44ae-a140-3b6d3e8702bc" containerName="istio-proxy" Aug 05 18:08:58.123922 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.123902 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:08:58.128824 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.128793 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 18:08:58.129214 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.129196 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:08:58.140189 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.140152 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t"] Aug 05 18:08:58.183609 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.183580 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t\" (UID: \"78367c42-4d2a-4f13-8c7e-eaf60a789ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:08:58.284644 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.284591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t\" (UID: \"78367c42-4d2a-4f13-8c7e-eaf60a789ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:08:58.285205 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.285185 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t\" (UID: \"78367c42-4d2a-4f13-8c7e-eaf60a789ca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:08:58.434176 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.434080 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:08:58.574770 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.574721 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t"] Aug 05 18:08:58.576944 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:08:58.576919 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78367c42_4d2a_4f13_8c7e_eaf60a789ca0.slice/crio-07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51 WatchSource:0}: Error finding container 07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51: Status 404 returned error can't find the container with id 07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51 Aug 05 18:08:58.675194 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:58.675162 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" event={"ID":"78367c42-4d2a-4f13-8c7e-eaf60a789ca0","Type":"ContainerStarted","Data":"07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51"} Aug 05 18:08:59.680752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:59.680659 2561 generic.go:356] "Generic (PLEG): container finished" podID="78367c42-4d2a-4f13-8c7e-eaf60a789ca0" containerID="31f35bfe50eb262c9caff707841f5763d8e3a5d3c677197f0ca741fedddf49c3" exitCode=0 Aug 05 18:08:59.680752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:08:59.680701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" event={"ID":"78367c42-4d2a-4f13-8c7e-eaf60a789ca0","Type":"ContainerDied","Data":"31f35bfe50eb262c9caff707841f5763d8e3a5d3c677197f0ca741fedddf49c3"} Aug 05 18:09:00.812658 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:00.812626 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:09:00.908083 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:00.908040 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca\") pod \"78367c42-4d2a-4f13-8c7e-eaf60a789ca0\" (UID: \"78367c42-4d2a-4f13-8c7e-eaf60a789ca0\") " Aug 05 18:09:00.908459 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:00.908435 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78367c42-4d2a-4f13-8c7e-eaf60a789ca0" (UID: "78367c42-4d2a-4f13-8c7e-eaf60a789ca0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:09:01.009437 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:01.009400 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78367c42-4d2a-4f13-8c7e-eaf60a789ca0-maas-api-service-ca\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:01.690845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:01.690809 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" event={"ID":"78367c42-4d2a-4f13-8c7e-eaf60a789ca0","Type":"ContainerDied","Data":"07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51"} Aug 05 18:09:01.690845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:01.690835 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t" Aug 05 18:09:01.691208 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:01.690858 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07072ed73d0f06902f8a0d3bb747373544d0a14adfc4a03592e6ddc9f86b5c51" Aug 05 18:09:08.873203 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:08.873165 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t"] Aug 05 18:09:08.876097 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:08.876072 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e99d153-jzz5t"] Aug 05 18:09:10.068743 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:10.068698 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78367c42-4d2a-4f13-8c7e-eaf60a789ca0" path="/var/lib/kubelet/pods/78367c42-4d2a-4f13-8c7e-eaf60a789ca0/volumes" Aug 05 18:09:10.904886 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:10.904854 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:09:16.276336 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:16.276301 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:09:16.276769 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:16.276527 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" containerID="cri-o://54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7" gracePeriod=30 Aug 05 18:09:21.526026 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.525997 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:09:21.576646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576541 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576579 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576602 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576638 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576655 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576675 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576743 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576781 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z886\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576821 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs\") pod \"9768c2dd-3afc-4e32-83e3-376e6af1345d\" (UID: \"9768c2dd-3afc-4e32-83e3-376e6af1345d\") " Aug 05 18:09:21.576991 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.576966 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:09:21.577327 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.577055 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.577327 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.577168 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:09:21.577327 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.577204 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:09:21.577516 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.577495 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:09:21.577789 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.577752 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:09:21.579043 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.579006 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:09:21.579558 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.579531 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886" (OuterVolumeSpecName: "kube-api-access-8z886") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "kube-api-access-8z886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:09:21.579638 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.579566 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:09:21.579638 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.579578 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9768c2dd-3afc-4e32-83e3-376e6af1345d" (UID: "9768c2dd-3afc-4e32-83e3-376e6af1345d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:09:21.677571 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677524 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677571 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677563 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677571 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677575 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677825 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677585 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677825 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677598 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768c2dd-3afc-4e32-83e3-376e6af1345d-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677825 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677609 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768c2dd-3afc-4e32-83e3-376e6af1345d-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677825 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677620 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768c2dd-3afc-4e32-83e3-376e6af1345d-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.677825 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.677631 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z886\" (UniqueName: \"kubernetes.io/projected/9768c2dd-3afc-4e32-83e3-376e6af1345d-kube-api-access-8z886\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:09:21.772827 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.772788 2561 generic.go:356] "Generic (PLEG): container finished" podID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerID="54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7" exitCode=0 Aug 05 18:09:21.773006 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.772843 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" Aug 05 18:09:21.773006 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.772876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerDied","Data":"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7"} Aug 05 18:09:21.773006 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.772934 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr" event={"ID":"9768c2dd-3afc-4e32-83e3-376e6af1345d","Type":"ContainerDied","Data":"a32fcf3373e4d25bdae7745441837a0be58856e08261f38c14cb8864bab31a89"} Aug 05 18:09:21.773006 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.772966 2561 scope.go:117] "RemoveContainer" containerID="54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7" Aug 05 18:09:21.781778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.781759 2561 scope.go:117] "RemoveContainer" containerID="1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4" Aug 05 18:09:21.789844 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.789823 2561 scope.go:117] "RemoveContainer" containerID="54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7" Aug 05 18:09:21.790137 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:09:21.790115 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7\": container with ID starting with 54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7 not found: ID does not exist" containerID="54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7" Aug 05 18:09:21.790189 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.790148 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7"} err="failed to get container status \"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7\": rpc error: code = NotFound desc = could not find container \"54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7\": container with ID starting with 54b1c6c14b56f1d5a0db11173d888931599646f78453b4a9740acc88587056e7 not found: ID does not exist" Aug 05 18:09:21.790189 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.790168 2561 scope.go:117] "RemoveContainer" containerID="1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4" Aug 05 18:09:21.790485 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:09:21.790463 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4\": container with ID starting with 1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4 not found: ID does not exist" containerID="1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4" Aug 05 18:09:21.790546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.790497 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4"} err="failed to get container status \"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4\": rpc error: code = NotFound desc = could not find container \"1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4\": container with ID starting with 1512ea168cd609b426f0cfe86469577b41c5f2af29d58b11ba5d65dce349eef4 not found: ID does not exist" Aug 05 18:09:21.800203 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.800169 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:09:21.802926 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:21.802898 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2e99d153-openshift-default-6878c5d45b-vrhhr"] Aug 05 18:09:22.068573 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:22.068531 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" path="/var/lib/kubelet/pods/9768c2dd-3afc-4e32-83e3-376e6af1345d/volumes" Aug 05 18:09:50.407682 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:09:50.407643 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:11:33.475243 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475205 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475506 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475519 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475532 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78367c42-4d2a-4f13-8c7e-eaf60a789ca0" containerName="revoke-keys" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475538 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78367c42-4d2a-4f13-8c7e-eaf60a789ca0" containerName="revoke-keys" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475593 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475600 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:11:33.475702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.475605 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78367c42-4d2a-4f13-8c7e-eaf60a789ca0" containerName="revoke-keys" Aug 05 18:11:33.478399 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.478382 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.480967 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.480944 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9e3b0986-openshift-default-dockercfg-mtd2l\"" Aug 05 18:11:33.493838 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.493813 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:11:33.592593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592596 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztt7\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592615 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592642 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592698 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592729 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592766 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.592841 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.592782 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694011 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztt7\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694051 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694078 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694189 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694622 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694552 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694667 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694647 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694722 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694369 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694755 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694778 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694678 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694875 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694792 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.694987 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.694970 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.696771 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.696747 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.696891 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.696874 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.702788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.702762 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztt7\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.703166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.703149 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token\") pod \"e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.787969 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.787937 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:33.924906 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.924876 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:11:33.927413 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:11:33.927374 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda985d818_068c_4861_93e7_9ffc9dac29b7.slice/crio-bcddb8dbc76d1014ae2318427bfaf4ac722e72dff90969681678de3d2bf86042 WatchSource:0}: Error finding container bcddb8dbc76d1014ae2318427bfaf4ac722e72dff90969681678de3d2bf86042: Status 404 returned error can't find the container with id bcddb8dbc76d1014ae2318427bfaf4ac722e72dff90969681678de3d2bf86042 Aug 05 18:11:33.929397 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.929363 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:33.929499 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.929446 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:33.929499 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:33.929489 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:34.308740 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:34.308700 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerStarted","Data":"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34"} Aug 05 18:11:34.308929 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:34.308785 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerStarted","Data":"bcddb8dbc76d1014ae2318427bfaf4ac722e72dff90969681678de3d2bf86042"} Aug 05 18:11:34.788552 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:34.788513 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:34.793513 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:34.793489 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:34.820738 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:34.820684 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podStartSLOduration=1.820666678 podStartE2EDuration="1.820666678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:11:34.337782748 +0000 UTC m=+2320.814065214" watchObservedRunningTime="2026-08-05 18:11:34.820666678 +0000 UTC m=+2321.296949143" Aug 05 18:11:35.312577 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:35.312538 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:35.313549 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:35.313531 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:36.305748 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:36.305714 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:11:45.210148 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:45.210057 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:11:51.354430 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:51.354385 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 05 18:11:51.354821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:51.354452 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 05 18:11:52.381253 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.381126 2561 generic.go:356] "Generic (PLEG): container finished" podID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerID="8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34" exitCode=0 Aug 05 18:11:52.381253 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.381163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerDied","Data":"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34"} Aug 05 18:11:52.381700 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.381532 2561 scope.go:117] "RemoveContainer" containerID="8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34" Aug 05 18:11:52.382498 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.382465 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:52.382611 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.382554 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:52.382675 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:52.382619 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:11:53.389375 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:53.389334 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerStarted","Data":"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f"} Aug 05 18:11:53.788828 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:53.788788 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:53.789140 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:53.788818 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Aug 05 18:11:53.789353 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:53.789328 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Aug 05 18:11:54.788916 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:54.788871 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Aug 05 18:11:54.789574 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:54.789537 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Aug 05 18:11:55.789369 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:55.789306 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Aug 05 18:11:55.789874 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:55.789424 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Aug 05 18:11:56.789157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:56.788684 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Aug 05 18:11:56.789157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:56.788768 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Aug 05 18:11:57.789117 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:57.789082 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Aug 05 18:11:57.789421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:57.789392 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Aug 05 18:11:58.792804 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:58.792768 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:58.793245 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:58.793045 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:11:58.793765 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:11:58.793745 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:12:07.224544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.224503 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:12:07.225691 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.225664 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:12:07.225691 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.225693 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9768c2dd-3afc-4e32-83e3-376e6af1345d" containerName="istio-proxy" Aug 05 18:12:07.229076 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.229050 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.232508 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.232483 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9e3b0986\"/\"kube-root-ca.crt\"" Aug 05 18:12:07.232625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.232483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9e3b0986\"/\"e2e-lifecycle-model-9e3b0986-kserve-self-signed-certs\"" Aug 05 18:12:07.232625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.232486 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9e3b0986\"/\"openshift-service-ca.crt\"" Aug 05 18:12:07.232625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.232540 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9e3b0986\"/\"default-dockercfg-6zbfr\"" Aug 05 18:12:07.238307 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.238270 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:12:07.287319 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287288 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.287500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.287500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.287500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbhq\" (UniqueName: \"kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.287500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287419 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.287500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.287437 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396362 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbhq\" (UniqueName: \"kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396429 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.396460 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.399947 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.400277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.400511 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.402925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.402801 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.405614 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.405556 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.411849 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.411819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbhq\" (UniqueName: \"kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq\") pod \"e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.539211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.539167 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:07.678061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:07.677993 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:12:07.680124 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:12:07.680097 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb28cc12a_c5b3_4c77_9066_50e9ea17d06d.slice/crio-760d5ba676dd1871f1c22712fa1b6395a94610480611c838af4155f79d4e21f2 WatchSource:0}: Error finding container 760d5ba676dd1871f1c22712fa1b6395a94610480611c838af4155f79d4e21f2: Status 404 returned error can't find the container with id 760d5ba676dd1871f1c22712fa1b6395a94610480611c838af4155f79d4e21f2 Aug 05 18:12:08.459908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:08.459873 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" event={"ID":"b28cc12a-c5b3-4c77-9066-50e9ea17d06d","Type":"ContainerStarted","Data":"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675"} Aug 05 18:12:08.460332 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:08.459918 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" event={"ID":"b28cc12a-c5b3-4c77-9066-50e9ea17d06d","Type":"ContainerStarted","Data":"760d5ba676dd1871f1c22712fa1b6395a94610480611c838af4155f79d4e21f2"} Aug 05 18:12:08.482650 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:08.482583 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" podStartSLOduration=1.482566316 podStartE2EDuration="1.482566316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:12:08.481262845 +0000 UTC m=+2354.957545315" watchObservedRunningTime="2026-08-05 18:12:08.482566316 +0000 UTC m=+2354.958848827" Aug 05 18:12:09.602759 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:09.602717 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:12:17.539888 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:17.539844 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:17.539888 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:17.539898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:17.552326 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:17.552302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:18.517702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:18.517672 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:12:25.100751 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:25.100712 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:12:48.797664 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:48.797625 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:12:54.087924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:54.087886 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:12:54.092962 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:54.092939 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:12:56.500734 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:56.500692 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:12:59.273984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:12:59.273946 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:13:14.293270 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.293205 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" podUID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" containerName="main" containerID="cri-o://0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675" gracePeriod=45 Aug 05 18:13:14.536682 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.536651 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:13:14.688045 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.687957 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688045 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688029 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688316 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688087 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdbhq\" (UniqueName: \"kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688316 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688121 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688316 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688162 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688335 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:14.688480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688342 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm\") pod \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\" (UID: \"b28cc12a-c5b3-4c77-9066-50e9ea17d06d\") " Aug 05 18:13:14.688480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688413 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home" (OuterVolumeSpecName: "home") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:14.688480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688432 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:14.688668 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688648 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:14.688712 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688673 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:14.688712 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.688687 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:14.690458 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.690420 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:13:14.690569 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.690455 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm" (OuterVolumeSpecName: "dshm") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:14.690569 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.690468 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq" (OuterVolumeSpecName: "kube-api-access-sdbhq") pod "b28cc12a-c5b3-4c77-9066-50e9ea17d06d" (UID: "b28cc12a-c5b3-4c77-9066-50e9ea17d06d"). InnerVolumeSpecName "kube-api-access-sdbhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:14.743622 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.743546 2561 generic.go:356] "Generic (PLEG): container finished" podID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" containerID="0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675" exitCode=0 Aug 05 18:13:14.743622 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.743588 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" event={"ID":"b28cc12a-c5b3-4c77-9066-50e9ea17d06d","Type":"ContainerDied","Data":"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675"} Aug 05 18:13:14.743622 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.743593 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" Aug 05 18:13:14.743622 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.743615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww" event={"ID":"b28cc12a-c5b3-4c77-9066-50e9ea17d06d","Type":"ContainerDied","Data":"760d5ba676dd1871f1c22712fa1b6395a94610480611c838af4155f79d4e21f2"} Aug 05 18:13:14.743908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.743631 2561 scope.go:117] "RemoveContainer" containerID="0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675" Aug 05 18:13:14.752428 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.752410 2561 scope.go:117] "RemoveContainer" containerID="0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675" Aug 05 18:13:14.752729 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:13:14.752708 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675\": container with ID starting with 0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675 not found: ID does not exist" containerID="0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675" Aug 05 18:13:14.752787 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.752738 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675"} err="failed to get container status \"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675\": rpc error: code = NotFound desc = could not find container \"0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675\": container with ID starting with 0c39967219a9d2fa11cd1ff267a197a411a78dea009d4629d47fdfa646e58675 not found: ID does not exist" Aug 05 18:13:14.765754 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.765722 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:13:14.769787 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.769759 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9e3b0986/e2e-lifecycle-model-9e3b0986-kserve-bd9b665ff-68qww"] Aug 05 18:13:14.789682 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.789653 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:14.789682 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.789679 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbhq\" (UniqueName: \"kubernetes.io/projected/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-kube-api-access-sdbhq\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:14.789809 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:14.789689 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28cc12a-c5b3-4c77-9066-50e9ea17d06d-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:16.068844 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:16.068811 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" path="/var/lib/kubelet/pods/b28cc12a-c5b3-4c77-9066-50e9ea17d06d/volumes" Aug 05 18:13:43.104929 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:43.104891 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:13:43.105488 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:43.105127 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" containerID="cri-o://e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f" gracePeriod=30 Aug 05 18:13:43.796026 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:43.795987 2561 patch_prober.go:28] interesting pod/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:13:43.796200 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:43.796046 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:13:45.113019 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:45.112984 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:13:48.347353 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.347322 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:13:48.458385 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458295 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458385 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458335 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458387 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458417 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458461 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458499 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pztt7\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458540 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458567 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.458686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458608 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo\") pod \"a985d818-068c-4861-93e7-9ffc9dac29b7\" (UID: \"a985d818-068c-4861-93e7-9ffc9dac29b7\") " Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458716 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458766 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458859 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458885 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.458983 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:13:48.459053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.459052 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.459556 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.459069 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.459556 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.459083 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.459556 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.459097 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.461048 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.461025 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:13:48.461150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.461047 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:13:48.461150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.461059 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:48.461150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.461077 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7" (OuterVolumeSpecName: "kube-api-access-pztt7") pod "a985d818-068c-4861-93e7-9ffc9dac29b7" (UID: "a985d818-068c-4861-93e7-9ffc9dac29b7"). InnerVolumeSpecName "kube-api-access-pztt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:13:48.559898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.559845 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a985d818-068c-4861-93e7-9ffc9dac29b7-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.559898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.559888 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.559898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.559898 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztt7\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-kube-api-access-pztt7\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.559898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.559907 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.559898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.559916 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a985d818-068c-4861-93e7-9ffc9dac29b7-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:13:48.887797 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.887765 2561 generic.go:356] "Generic (PLEG): container finished" podID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerID="e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f" exitCode=0 Aug 05 18:13:48.888023 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.887836 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" Aug 05 18:13:48.888023 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.887839 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerDied","Data":"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f"} Aug 05 18:13:48.888023 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.887956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb" event={"ID":"a985d818-068c-4861-93e7-9ffc9dac29b7","Type":"ContainerDied","Data":"bcddb8dbc76d1014ae2318427bfaf4ac722e72dff90969681678de3d2bf86042"} Aug 05 18:13:48.888023 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.887977 2561 scope.go:117] "RemoveContainer" containerID="e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f" Aug 05 18:13:48.896772 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.896754 2561 scope.go:117] "RemoveContainer" containerID="8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34" Aug 05 18:13:48.904689 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.904670 2561 scope.go:117] "RemoveContainer" containerID="e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f" Aug 05 18:13:48.904959 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:13:48.904940 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f\": container with ID starting with e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f not found: ID does not exist" containerID="e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f" Aug 05 18:13:48.905002 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.904969 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f"} err="failed to get container status \"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f\": rpc error: code = NotFound desc = could not find container \"e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f\": container with ID starting with e8dd45a837752b15fd0387468fb32754f82533208c9fc43ff46f6b2e0a731c9f not found: ID does not exist" Aug 05 18:13:48.905002 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.904988 2561 scope.go:117] "RemoveContainer" containerID="8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34" Aug 05 18:13:48.905206 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:13:48.905191 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34\": container with ID starting with 8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34 not found: ID does not exist" containerID="8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34" Aug 05 18:13:48.905266 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.905211 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34"} err="failed to get container status \"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34\": rpc error: code = NotFound desc = could not find container \"8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34\": container with ID starting with 8c651820791928ed5802c0915436f653c3cd5aafa5a01760ddd26984bc3b7c34 not found: ID does not exist" Aug 05 18:13:48.910507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.910486 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:13:48.919307 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:48.915822 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9e3b0986-openshift-default-67758f88-7jwlb"] Aug 05 18:13:50.072292 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:50.072256 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" path="/var/lib/kubelet/pods/a985d818-068c-4861-93e7-9ffc9dac29b7/volumes" Aug 05 18:13:57.404878 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:13:57.404843 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:14:05.613124 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:05.613088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:14:51.014430 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014354 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014675 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" containerName="main" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014689 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" containerName="main" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014697 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014702 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014715 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014721 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014765 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014773 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a985d818-068c-4861-93e7-9ffc9dac29b7" containerName="istio-proxy" Aug 05 18:14:51.014821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.014778 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28cc12a-c5b3-4c77-9066-50e9ea17d06d" containerName="main" Aug 05 18:14:51.017674 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.017658 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.020873 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.020852 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9261d9-openshift-default-dockercfg-7sv5h\"" Aug 05 18:14:51.035688 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.035662 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:14:51.067990 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.067958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.067999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068018 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zd5z\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068339 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068174 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068339 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068196 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068339 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068214 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.068339 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.068299 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169140 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169100 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169146 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169338 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169630 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zd5z\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169630 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169630 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169460 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169630 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169564 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169803 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169860 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169807 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.169901 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.169873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.170185 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.170150 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.171740 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.171721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.171936 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.171905 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.180404 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.180382 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.180626 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.180593 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zd5z\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z\") pod \"e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.327139 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.327050 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:51.471191 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.471160 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:14:51.473743 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:14:51.473710 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d17b424_1449_4bd2_a5aa_904275a774b9.slice/crio-c99a1b875430dd0e733a6112f436c548e8bdfa73737be762869e948ffdf25cf8 WatchSource:0}: Error finding container c99a1b875430dd0e733a6112f436c548e8bdfa73737be762869e948ffdf25cf8: Status 404 returned error can't find the container with id c99a1b875430dd0e733a6112f436c548e8bdfa73737be762869e948ffdf25cf8 Aug 05 18:14:51.475822 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.475787 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:51.475914 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.475855 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:51.475914 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:51.475882 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:14:52.149302 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:52.149261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerStarted","Data":"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5"} Aug 05 18:14:52.149709 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:52.149319 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerStarted","Data":"c99a1b875430dd0e733a6112f436c548e8bdfa73737be762869e948ffdf25cf8"} Aug 05 18:14:52.177406 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:52.177350 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podStartSLOduration=2.177336539 podStartE2EDuration="2.177336539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:14:52.17656231 +0000 UTC m=+2518.652844776" watchObservedRunningTime="2026-08-05 18:14:52.177336539 +0000 UTC m=+2518.653619075" Aug 05 18:14:52.328218 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:52.328175 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:52.333595 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:52.333558 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:53.153500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:53.153460 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:53.154566 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:53.154543 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:14:58.305652 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:14:58.305620 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:15:00.133374 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.133343 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp"] Aug 05 18:15:00.136344 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.136324 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" Aug 05 18:15:00.138851 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.138831 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 18:15:00.146852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.146828 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" Aug 05 18:15:00.146972 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.146833 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp"] Aug 05 18:15:00.314377 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:15:00.314343 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccca35ad_eb6a_4b8d_9953_34bebb261290.slice/crio-fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510 WatchSource:0}: Error finding container fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510: Status 404 returned error can't find the container with id fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510 Aug 05 18:15:00.316321 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:00.316191 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp"] Aug 05 18:15:01.188760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:01.188716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" event={"ID":"ccca35ad-eb6a-4b8d-9953-34bebb261290","Type":"ContainerStarted","Data":"6095184c5c356b47d584c25443dd6bec16686f19c76c14933d38facb37541318"} Aug 05 18:15:01.189291 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:01.188788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" event={"ID":"ccca35ad-eb6a-4b8d-9953-34bebb261290","Type":"ContainerStarted","Data":"fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510"} Aug 05 18:15:01.214658 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:01.214599 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" podStartSLOduration=1.214583333 podStartE2EDuration="1.214583333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:15:01.213013828 +0000 UTC m=+2527.689296295" watchObservedRunningTime="2026-08-05 18:15:01.214583333 +0000 UTC m=+2527.690865798" Aug 05 18:15:02.196493 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:02.196450 2561 generic.go:356] "Generic (PLEG): container finished" podID="ccca35ad-eb6a-4b8d-9953-34bebb261290" containerID="6095184c5c356b47d584c25443dd6bec16686f19c76c14933d38facb37541318" exitCode=0 Aug 05 18:15:02.197009 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:02.196587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" event={"ID":"ccca35ad-eb6a-4b8d-9953-34bebb261290","Type":"ContainerDied","Data":"6095184c5c356b47d584c25443dd6bec16686f19c76c14933d38facb37541318"} Aug 05 18:15:03.356713 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:03.356688 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" Aug 05 18:15:04.209724 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:04.209677 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" event={"ID":"ccca35ad-eb6a-4b8d-9953-34bebb261290","Type":"ContainerDied","Data":"fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510"} Aug 05 18:15:04.209911 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:04.209754 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2ae5ab21ca5d303b619ed4ebfcb56f2317a499b7b1e667ef09ff35d8c33510" Aug 05 18:15:04.209911 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:04.209829 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29765895-nqzcp" Aug 05 18:15:08.249046 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:08.249011 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:37618->10.132.0.33:15021: read: connection reset by peer" start-of-body= Aug 05 18:15:08.249537 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:08.249076 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": read tcp 10.132.0.2:37618->10.132.0.33:15021: read: connection reset by peer" Aug 05 18:15:09.232940 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.232905 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerID="9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5" exitCode=0 Aug 05 18:15:09.233127 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.232959 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerDied","Data":"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5"} Aug 05 18:15:09.233433 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.233412 2561 scope.go:117] "RemoveContainer" containerID="9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5" Aug 05 18:15:09.234377 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.234344 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:09.234442 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.234415 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:09.234442 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:09.234445 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:10.238753 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:10.238710 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerStarted","Data":"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89"} Aug 05 18:15:10.327188 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:10.327152 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:15:10.327463 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:10.327439 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 05 18:15:10.327529 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:10.327489 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 05 18:15:11.327899 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:11.327862 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 05 18:15:11.328325 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:11.327925 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 05 18:15:12.327444 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:12.327409 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 05 18:15:12.327655 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:12.327461 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 05 18:15:13.328191 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:13.328154 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 05 18:15:13.328605 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:13.328209 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 05 18:15:14.327853 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:14.327807 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body= Aug 05 18:15:14.328055 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:14.327873 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" Aug 05 18:15:15.331049 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:15.331019 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:15:15.331504 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:15.331261 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:15:15.332119 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:15.332105 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:15:24.855646 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.855611 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:15:24.856020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.855886 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccca35ad-eb6a-4b8d-9953-34bebb261290" containerName="cleanup" Aug 05 18:15:24.856020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.855897 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccca35ad-eb6a-4b8d-9953-34bebb261290" containerName="cleanup" Aug 05 18:15:24.856020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.855956 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccca35ad-eb6a-4b8d-9953-34bebb261290" containerName="cleanup" Aug 05 18:15:24.858039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.858023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.862656 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.862633 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-955c47-openshift-default-dockercfg-9w2l4\"" Aug 05 18:15:24.876059 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.876036 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:15:24.965968 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.965936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.965968 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.965976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966255 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966007 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966255 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966255 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966255 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966153 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966255 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966173 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966455 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhbj\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:24.966455 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:24.966321 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067446 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067415 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067638 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067638 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhbj\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067748 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067809 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.067899 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.067877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068086 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068065 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068168 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068106 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068168 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068379 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068190 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068379 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.068927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.068899 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.069062 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.069038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.069256 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.069209 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.070483 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.070461 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.071462 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.071440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.084503 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.084432 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.086146 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.086125 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhbj\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj\") pod \"e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.167463 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.167367 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:25.307592 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.307557 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:15:25.310463 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:15:25.310431 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be290cb_a3e4_4c55_8539_7be94154c8dd.slice/crio-07aaa99c6452b20916f123ef3dd96303722d1a462e2f83fb40e8d67c2fd0d109 WatchSource:0}: Error finding container 07aaa99c6452b20916f123ef3dd96303722d1a462e2f83fb40e8d67c2fd0d109: Status 404 returned error can't find the container with id 07aaa99c6452b20916f123ef3dd96303722d1a462e2f83fb40e8d67c2fd0d109 Aug 05 18:15:25.312603 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.312570 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:25.312695 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.312644 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:25.312695 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:25.312677 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:26.306968 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:26.306926 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerStarted","Data":"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229"} Aug 05 18:15:26.307480 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:26.306988 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerStarted","Data":"07aaa99c6452b20916f123ef3dd96303722d1a462e2f83fb40e8d67c2fd0d109"} Aug 05 18:15:26.331468 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:26.331416 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podStartSLOduration=2.331400444 podStartE2EDuration="2.331400444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:15:26.330402541 +0000 UTC m=+2552.806685040" watchObservedRunningTime="2026-08-05 18:15:26.331400444 +0000 UTC m=+2552.807682908" Aug 05 18:15:27.167749 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:27.167708 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:27.173183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:27.173155 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:27.310917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:27.310884 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:27.311969 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:27.311952 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:33.105309 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:33.105271 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:15:42.593924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:42.593885 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:34368->10.132.0.35:15021: read: connection reset by peer" start-of-body= Aug 05 18:15:42.594372 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:42.593961 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:34368->10.132.0.35:15021: read: connection reset by peer" Aug 05 18:15:43.377791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.377759 2561 generic.go:356] "Generic (PLEG): container finished" podID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerID="b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229" exitCode=0 Aug 05 18:15:43.377972 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.377838 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerDied","Data":"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229"} Aug 05 18:15:43.378212 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.378199 2561 scope.go:117] "RemoveContainer" containerID="b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229" Aug 05 18:15:43.379241 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.379194 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:43.379358 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.379303 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:43.379426 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:43.379373 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:15:44.383721 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:44.383686 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerStarted","Data":"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c"} Aug 05 18:15:45.168402 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:45.168362 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:45.168662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:45.168638 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Aug 05 18:15:45.168718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:45.168691 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Aug 05 18:15:46.167791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:46.167755 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Aug 05 18:15:46.168183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:46.167822 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Aug 05 18:15:47.167930 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:47.167896 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Aug 05 18:15:47.168338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:47.167977 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Aug 05 18:15:48.167902 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:48.167868 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Aug 05 18:15:48.168073 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:48.167935 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Aug 05 18:15:49.167851 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:49.167813 2561 patch_prober.go:28] interesting pod/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Aug 05 18:15:49.168037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:49.167876 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Aug 05 18:15:50.171596 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:50.171566 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:50.171981 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:50.171865 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:50.172594 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:50.172577 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:15:54.185477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:15:54.185444 2561 scope.go:117] "RemoveContainer" containerID="31f35bfe50eb262c9caff707841f5763d8e3a5d3c677197f0ca741fedddf49c3" Aug 05 18:16:08.893981 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:08.893949 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:16:15.795471 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:15.795381 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:16:24.232974 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.232932 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:16:24.235745 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.235723 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.239205 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.239179 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-955c47\"/\"default-dockercfg-r5p57\"" Aug 05 18:16:24.239394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.239250 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-955c47\"/\"kube-root-ca.crt\"" Aug 05 18:16:24.239394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.239297 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-955c47\"/\"test-model-955c47-kserve-self-signed-certs\"" Aug 05 18:16:24.239394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.239307 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-955c47\"/\"openshift-service-ca.crt\"" Aug 05 18:16:24.248561 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.248529 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:16:24.381869 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.381828 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.382404 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.382369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.382567 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.382442 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pjh\" (UniqueName: \"kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.382567 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.382549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.382800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.382591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.382800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.382651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.483928 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483830 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.483928 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.483928 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483917 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pjh\" (UniqueName: \"kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484223 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484223 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484223 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.483989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484387 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.484332 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484387 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.484364 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.484457 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.484411 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.486446 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.486427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.486700 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.486682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.492677 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.492650 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pjh\" (UniqueName: \"kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh\") pod \"test-model-955c47-kserve-6f4d7f8c9-nlf9t\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.546016 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.545977 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:24.685207 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:24.685009 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:16:24.688024 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:16:24.687998 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a807070_cded_4f3c_b5af_d6fef78b96a0.slice/crio-f608f725c8c166613a42186e3e25ac744eb5cd97b113aca3dca19b9507b60d4c WatchSource:0}: Error finding container f608f725c8c166613a42186e3e25ac744eb5cd97b113aca3dca19b9507b60d4c: Status 404 returned error can't find the container with id f608f725c8c166613a42186e3e25ac744eb5cd97b113aca3dca19b9507b60d4c Aug 05 18:16:25.568817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:25.568781 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" event={"ID":"6a807070-cded-4f3c-b5af-d6fef78b96a0","Type":"ContainerStarted","Data":"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef"} Aug 05 18:16:25.568817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:25.568826 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" event={"ID":"6a807070-cded-4f3c-b5af-d6fef78b96a0","Type":"ContainerStarted","Data":"f608f725c8c166613a42186e3e25ac744eb5cd97b113aca3dca19b9507b60d4c"} Aug 05 18:16:25.587897 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:25.587846 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" podStartSLOduration=1.587831729 podStartE2EDuration="1.587831729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:16:25.586611504 +0000 UTC m=+2612.062893969" watchObservedRunningTime="2026-08-05 18:16:25.587831729 +0000 UTC m=+2612.064114224" Aug 05 18:16:28.741897 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:28.741854 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:16:34.546702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:34.546664 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:34.547098 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:34.546753 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:34.559118 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:34.559096 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:16:34.616430 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:16:34.616403 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:17:08.518625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:08.518591 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:17:11.213923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:11.213885 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:17:21.700946 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:21.700906 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:17:23.537832 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.537774 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" podUID="6a807070-cded-4f3c-b5af-d6fef78b96a0" containerName="main" containerID="cri-o://2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef" gracePeriod=45 Aug 05 18:17:23.787336 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.787311 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:17:23.797756 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797691 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.797756 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797728 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5pjh\" (UniqueName: \"kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.797945 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797770 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.797945 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797808 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.797945 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797855 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.797945 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797899 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home\") pod \"6a807070-cded-4f3c-b5af-d6fef78b96a0\" (UID: \"6a807070-cded-4f3c-b5af-d6fef78b96a0\") " Aug 05 18:17:23.798150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.797943 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:23.798150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.798113 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:23.798256 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.798196 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:23.798256 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.798196 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home" (OuterVolumeSpecName: "home") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:23.798347 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.798213 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:23.800124 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.800101 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm" (OuterVolumeSpecName: "dshm") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:23.800453 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.800428 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:17:23.800453 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.800439 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh" (OuterVolumeSpecName: "kube-api-access-l5pjh") pod "6a807070-cded-4f3c-b5af-d6fef78b96a0" (UID: "6a807070-cded-4f3c-b5af-d6fef78b96a0"). InnerVolumeSpecName "kube-api-access-l5pjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:23.818908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.818874 2561 generic.go:356] "Generic (PLEG): container finished" podID="6a807070-cded-4f3c-b5af-d6fef78b96a0" containerID="2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef" exitCode=0 Aug 05 18:17:23.819061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.818917 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" Aug 05 18:17:23.819061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.818949 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" event={"ID":"6a807070-cded-4f3c-b5af-d6fef78b96a0","Type":"ContainerDied","Data":"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef"} Aug 05 18:17:23.819061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.818995 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t" event={"ID":"6a807070-cded-4f3c-b5af-d6fef78b96a0","Type":"ContainerDied","Data":"f608f725c8c166613a42186e3e25ac744eb5cd97b113aca3dca19b9507b60d4c"} Aug 05 18:17:23.819061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.819013 2561 scope.go:117] "RemoveContainer" containerID="2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef" Aug 05 18:17:23.828131 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.828114 2561 scope.go:117] "RemoveContainer" containerID="2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef" Aug 05 18:17:23.828471 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:17:23.828447 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef\": container with ID starting with 2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef not found: ID does not exist" containerID="2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef" Aug 05 18:17:23.828590 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.828485 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef"} err="failed to get container status \"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef\": rpc error: code = NotFound desc = could not find container \"2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef\": container with ID starting with 2281f1eea56a975bfd0d2ed5ae990d55781d0b08adaaf4296221ab3114b187ef not found: ID does not exist" Aug 05 18:17:23.840687 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.840662 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:17:23.844123 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.844102 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-955c47/test-model-955c47-kserve-6f4d7f8c9-nlf9t"] Aug 05 18:17:23.899643 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.899614 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5pjh\" (UniqueName: \"kubernetes.io/projected/6a807070-cded-4f3c-b5af-d6fef78b96a0-kube-api-access-l5pjh\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:23.899643 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.899638 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a807070-cded-4f3c-b5af-d6fef78b96a0-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:23.899643 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.899649 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:23.899814 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:23.899656 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a807070-cded-4f3c-b5af-d6fef78b96a0-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:24.068969 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:24.068875 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a807070-cded-4f3c-b5af-d6fef78b96a0" path="/var/lib/kubelet/pods/6a807070-cded-4f3c-b5af-d6fef78b96a0/volumes" Aug 05 18:17:26.266961 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:26.266929 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:17:26.267397 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:26.267141 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" containerID="cri-o://aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89" gracePeriod=30 Aug 05 18:17:30.334143 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:30.334106 2561 patch_prober.go:28] interesting pod/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:17:30.334566 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:30.334178 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:17:31.508815 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.508789 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:17:31.563629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563553 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563629 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563611 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563638 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563666 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563700 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563726 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563756 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.563852 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563817 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.564152 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.563855 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zd5z\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z\") pod \"7d17b424-1449-4bd2-a5aa-904275a774b9\" (UID: \"7d17b424-1449-4bd2-a5aa-904275a774b9\") " Aug 05 18:17:31.564152 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564043 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:31.564288 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564169 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.564288 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564184 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:17:31.564505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564434 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:31.564505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564451 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:31.564505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.564334 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:31.565811 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.565790 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:31.566183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.566160 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z" (OuterVolumeSpecName: "kube-api-access-7zd5z") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "kube-api-access-7zd5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:31.566483 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.566462 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:17:31.566483 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.566469 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d17b424-1449-4bd2-a5aa-904275a774b9" (UID: "7d17b424-1449-4bd2-a5aa-904275a774b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:31.664871 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664835 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.664871 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664861 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.664871 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664870 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.664871 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664879 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d17b424-1449-4bd2-a5aa-904275a774b9-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.665108 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664886 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.665108 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664894 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.665108 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664902 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zd5z\" (UniqueName: \"kubernetes.io/projected/7d17b424-1449-4bd2-a5aa-904275a774b9-kube-api-access-7zd5z\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.665108 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.664911 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d17b424-1449-4bd2-a5aa-904275a774b9-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:31.854378 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.854282 2561 generic.go:356] "Generic (PLEG): container finished" podID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerID="aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89" exitCode=0 Aug 05 18:17:31.854378 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.854332 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" Aug 05 18:17:31.854378 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.854356 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerDied","Data":"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89"} Aug 05 18:17:31.854632 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.854396 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2" event={"ID":"7d17b424-1449-4bd2-a5aa-904275a774b9","Type":"ContainerDied","Data":"c99a1b875430dd0e733a6112f436c548e8bdfa73737be762869e948ffdf25cf8"} Aug 05 18:17:31.854632 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.854419 2561 scope.go:117] "RemoveContainer" containerID="aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89" Aug 05 18:17:31.863511 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.863494 2561 scope.go:117] "RemoveContainer" containerID="9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5" Aug 05 18:17:31.876674 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.876644 2561 scope.go:117] "RemoveContainer" containerID="aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89" Aug 05 18:17:31.876810 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.876786 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:17:31.876983 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:17:31.876962 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89\": container with ID starting with aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89 not found: ID does not exist" containerID="aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89" Aug 05 18:17:31.877056 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.876996 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89"} err="failed to get container status \"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89\": rpc error: code = NotFound desc = could not find container \"aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89\": container with ID starting with aaa3fd267f92f2a4ab74752741f6e5cff1c4b71ce175de8c296bd93913d55d89 not found: ID does not exist" Aug 05 18:17:31.877056 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.877023 2561 scope.go:117] "RemoveContainer" containerID="9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5" Aug 05 18:17:31.877318 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:17:31.877300 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5\": container with ID starting with 9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5 not found: ID does not exist" containerID="9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5" Aug 05 18:17:31.877375 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.877325 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5"} err="failed to get container status \"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5\": rpc error: code = NotFound desc = could not find container \"9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5\": container with ID starting with 9f2a132f01ac4c6850784c04fc10d03a74b7b7384afaee7d4f0ce57fccb052f5 not found: ID does not exist" Aug 05 18:17:31.882813 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:31.882791 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9261d9-openshift-default-6fb956f87b-ttff2"] Aug 05 18:17:32.068600 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:32.068572 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" path="/var/lib/kubelet/pods/7d17b424-1449-4bd2-a5aa-904275a774b9/volumes" Aug 05 18:17:39.297955 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:39.297913 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:17:44.089111 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.089077 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:17:44.089569 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.089320 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" containerID="cri-o://a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c" gracePeriod=30 Aug 05 18:17:44.897398 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897294 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:17:44.897648 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897632 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a807070-cded-4f3c-b5af-d6fef78b96a0" containerName="main" Aug 05 18:17:44.897718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897651 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a807070-cded-4f3c-b5af-d6fef78b96a0" containerName="main" Aug 05 18:17:44.897718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897665 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:17:44.897718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897671 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:17:44.897885 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897732 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:17:44.897885 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897740 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:17:44.897885 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.897748 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a807070-cded-4f3c-b5af-d6fef78b96a0" containerName="main" Aug 05 18:17:44.902056 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.902032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.904514 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.904486 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-acb45a-openshift-default-dockercfg-2rgcs\"" Aug 05 18:17:44.914568 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.914533 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:17:44.976463 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976429 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976693 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976693 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976489 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskv9\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976693 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976693 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976631 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976693 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976670 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:44.976917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:44.976854 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078301 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078301 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078302 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078525 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskv9\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078693 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.078788 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078730 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.079024 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.078935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.079062 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.079040 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.079166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.079147 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.081049 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.081016 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.081200 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.081173 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.086417 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.086389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.086523 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.086428 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskv9\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9\") pod \"e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.213097 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.213017 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:45.353791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.353747 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:17:45.357135 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:17:45.357105 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dd1708_d2f1_489c_ae04_ec3e1df8b6f2.slice/crio-9fb50b83e661326cdfd1d09719824819cbd4de3fd4f2c24d98614a0be87cb468 WatchSource:0}: Error finding container 9fb50b83e661326cdfd1d09719824819cbd4de3fd4f2c24d98614a0be87cb468: Status 404 returned error can't find the container with id 9fb50b83e661326cdfd1d09719824819cbd4de3fd4f2c24d98614a0be87cb468 Aug 05 18:17:45.359745 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.359709 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:17:45.359850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.359795 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:17:45.359850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.359841 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:17:45.915204 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.915169 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerStarted","Data":"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb"} Aug 05 18:17:45.915417 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.915237 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerStarted","Data":"9fb50b83e661326cdfd1d09719824819cbd4de3fd4f2c24d98614a0be87cb468"} Aug 05 18:17:45.937865 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:45.937804 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podStartSLOduration=1.937786339 podStartE2EDuration="1.937786339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:17:45.936126743 +0000 UTC m=+2692.412409214" watchObservedRunningTime="2026-08-05 18:17:45.937786339 +0000 UTC m=+2692.414068803" Aug 05 18:17:46.213300 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:46.213202 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:46.218242 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:46.218201 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:46.919140 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:46.919109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:46.920270 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:46.920245 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:17:49.349580 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.349545 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:17:49.417655 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417553 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.417655 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417588 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zhbj\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.417655 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417618 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.417655 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417650 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417676 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417697 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417735 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417765 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417808 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo\") pod \"0be290cb-a3e4-4c55-8539-7be94154c8dd\" (UID: \"0be290cb-a3e4-4c55-8539-7be94154c8dd\") " Aug 05 18:17:49.418003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.417886 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:49.418320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418010 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:49.418320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418024 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:49.418320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418162 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.418320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418182 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.418320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418242 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.418742 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418367 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:49.418742 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.418387 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:17:49.420086 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.420054 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:49.420193 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.420106 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj" (OuterVolumeSpecName: "kube-api-access-5zhbj") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "kube-api-access-5zhbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:17:49.420570 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.420543 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:17:49.420660 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.420596 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0be290cb-a3e4-4c55-8539-7be94154c8dd" (UID: "0be290cb-a3e4-4c55-8539-7be94154c8dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:17:49.518717 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518688 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhbj\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-kube-api-access-5zhbj\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.518717 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518714 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be290cb-a3e4-4c55-8539-7be94154c8dd-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.518717 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518724 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.518717 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518731 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.518984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518739 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.518984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.518747 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be290cb-a3e4-4c55-8539-7be94154c8dd-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:17:49.938577 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.938538 2561 generic.go:356] "Generic (PLEG): container finished" podID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerID="a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c" exitCode=0 Aug 05 18:17:49.938760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.938599 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" Aug 05 18:17:49.938760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.938627 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerDied","Data":"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c"} Aug 05 18:17:49.938760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.938703 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz" event={"ID":"0be290cb-a3e4-4c55-8539-7be94154c8dd","Type":"ContainerDied","Data":"07aaa99c6452b20916f123ef3dd96303722d1a462e2f83fb40e8d67c2fd0d109"} Aug 05 18:17:49.938760 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.938726 2561 scope.go:117] "RemoveContainer" containerID="a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c" Aug 05 18:17:49.948331 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.948303 2561 scope.go:117] "RemoveContainer" containerID="b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229" Aug 05 18:17:49.956654 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.956632 2561 scope.go:117] "RemoveContainer" containerID="a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c" Aug 05 18:17:49.956996 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:17:49.956974 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c\": container with ID starting with a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c not found: ID does not exist" containerID="a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c" Aug 05 18:17:49.957074 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.957010 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c"} err="failed to get container status \"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c\": rpc error: code = NotFound desc = could not find container \"a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c\": container with ID starting with a01f792a614760de20292c19cf9a4dbef255caeae52cf24f65c6478012267d1c not found: ID does not exist" Aug 05 18:17:49.957074 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.957037 2561 scope.go:117] "RemoveContainer" containerID="b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229" Aug 05 18:17:49.957320 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:17:49.957294 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229\": container with ID starting with b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229 not found: ID does not exist" containerID="b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229" Aug 05 18:17:49.957368 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.957331 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229"} err="failed to get container status \"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229\": rpc error: code = NotFound desc = could not find container \"b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229\": container with ID starting with b96b582a06b3a9b1c0e6fac8dd18616a6693d3b818028c5abc7711433bf5d229 not found: ID does not exist" Aug 05 18:17:49.962738 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.962711 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:17:49.966477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:49.966453 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-955c47-openshift-default-5844ff7d5d-8wpbz"] Aug 05 18:17:50.068196 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:50.068165 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" path="/var/lib/kubelet/pods/0be290cb-a3e4-4c55-8539-7be94154c8dd/volumes" Aug 05 18:17:52.113759 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:52.113723 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:17:54.122568 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:54.122527 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:17:54.125364 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:17:54.125345 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:18:01.920079 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.920046 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:01.920456 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.920105 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:01.998113 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.998080 2561 generic.go:356] "Generic (PLEG): container finished" podID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerID="2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb" exitCode=0 Aug 05 18:18:01.998300 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.998159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerDied","Data":"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb"} Aug 05 18:18:01.998599 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.998583 2561 scope.go:117] "RemoveContainer" containerID="2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb" Aug 05 18:18:01.999697 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.999667 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:01.999747 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.999739 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:01.999790 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:01.999776 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:03.008391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:03.008352 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerStarted","Data":"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91"} Aug 05 18:18:03.213628 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:03.213586 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:18:03.213821 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:03.213798 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:03.213884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:03.213866 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:04.214376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:04.214334 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:04.214813 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:04.214392 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:05.214326 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:05.214289 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:05.214515 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:05.214348 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:06.213870 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:06.213834 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:06.214060 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:06.213888 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:07.213818 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:07.213773 2561 patch_prober.go:28] interesting pod/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Aug 05 18:18:07.214203 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:07.213839 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Aug 05 18:18:08.218541 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:08.218510 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:18:08.219047 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:08.218977 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:18:08.219600 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:08.219578 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:18:13.684062 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684029 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684363 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684376 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684392 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684401 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d17b424-1449-4bd2-a5aa-904275a774b9" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684412 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684417 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684471 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.684554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.684481 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be290cb-a3e4-4c55-8539-7be94154c8dd" containerName="istio-proxy" Aug 05 18:18:13.687590 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.687574 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.691445 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.691423 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-87a594-openshift-default-dockercfg-9dhlf\"" Aug 05 18:18:13.706058 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.706033 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:18:13.727611 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727580 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727742 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727742 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727653 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727753 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727856 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727856 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps6w\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727946 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.727946 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.728047 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.727976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828384 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828340 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828591 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828591 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828591 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828454 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828722 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps6w\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.828791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.829045 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828907 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.829045 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828955 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.829045 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.828992 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.829394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.829089 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.829394 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.829152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.831089 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.831061 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.831188 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.831094 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.837474 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.837450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.837809 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.837784 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps6w\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w\") pod \"e2e-api-b-87a594-openshift-default-69887dffbb-njts4\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:13.997815 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:13.997782 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:14.154752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:14.154724 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:18:14.156838 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:18:14.156810 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c20020_8e62_4876_ad65_53c433578932.slice/crio-c8ddcefc3b3552ab62c6ccfe048b2e531458c0fad21465b6581bd761c8d03475 WatchSource:0}: Error finding container c8ddcefc3b3552ab62c6ccfe048b2e531458c0fad21465b6581bd761c8d03475: Status 404 returned error can't find the container with id c8ddcefc3b3552ab62c6ccfe048b2e531458c0fad21465b6581bd761c8d03475 Aug 05 18:18:14.159451 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:14.159415 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:14.159534 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:14.159485 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:14.159534 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:14.159517 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:15.063739 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:15.063701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerStarted","Data":"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff"} Aug 05 18:18:15.064121 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:15.063750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerStarted","Data":"c8ddcefc3b3552ab62c6ccfe048b2e531458c0fad21465b6581bd761c8d03475"} Aug 05 18:18:15.086597 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:15.086538 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podStartSLOduration=2.086520868 podStartE2EDuration="2.086520868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:18:15.086168979 +0000 UTC m=+2721.562451444" watchObservedRunningTime="2026-08-05 18:18:15.086520868 +0000 UTC m=+2721.562803334" Aug 05 18:18:15.998299 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:15.998262 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:16.003708 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:16.003678 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:16.068644 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:16.068613 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:16.069011 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:16.068682 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:20.800700 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:20.800665 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:18:31.069592 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.069557 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:31.070028 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.069616 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:31.149107 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.149074 2561 generic.go:356] "Generic (PLEG): container finished" podID="a7c20020-8e62-4876-ad65-53c433578932" containerID="5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff" exitCode=0 Aug 05 18:18:31.149107 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.149115 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerDied","Data":"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff"} Aug 05 18:18:31.149534 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.149519 2561 scope.go:117] "RemoveContainer" containerID="5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff" Aug 05 18:18:31.150588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.150554 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:31.150660 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.150641 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:31.150699 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:31.150686 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:18:32.155496 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:32.155447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerStarted","Data":"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b"} Aug 05 18:18:32.999036 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:32.998989 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:32.999286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:32.999134 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:32.999286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:32.999183 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:33.998834 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:33.998793 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:33.999451 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:33.999418 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:34.998492 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:34.998446 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:34.998720 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:34.998530 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:35.999540 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:35.998984 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:35.999540 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:35.999068 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:36.999304 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:36.999254 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Aug 05 18:18:36.999534 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:36.999343 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Aug 05 18:18:38.002264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:38.002213 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:38.002647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:38.002598 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:18:38.003374 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:18:38.003351 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:19:08.308600 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:08.308560 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:19:12.477937 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:12.477901 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:19:12.478342 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:12.478177 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" containerID="cri-o://b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91" gracePeriod=30 Aug 05 18:19:17.719938 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.719910 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:19:17.779907 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.779873 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780090 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.779925 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780090 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.779971 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780090 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780002 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780090 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780027 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780317 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780096 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fskv9\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780317 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780125 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780317 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780165 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780317 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780203 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket\") pod \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\" (UID: \"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2\") " Aug 05 18:19:17.780516 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780353 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:19:17.780579 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780469 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:17.780579 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780540 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:17.780696 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780587 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.780696 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780605 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.780796 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780704 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:17.780796 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.780720 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:17.782552 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.782522 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:19:17.782552 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.782533 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9" (OuterVolumeSpecName: "kube-api-access-fskv9") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "kube-api-access-fskv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:19:17.782838 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.782822 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:17.782941 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.782925 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" (UID: "f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:19:17.881733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881696 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881726 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881735 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881744 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881753 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881760 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:17.881947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:17.881768 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskv9\" (UniqueName: \"kubernetes.io/projected/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2-kube-api-access-fskv9\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:18.348505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.348418 2561 generic.go:356] "Generic (PLEG): container finished" podID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerID="b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91" exitCode=0 Aug 05 18:19:18.348505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.348463 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerDied","Data":"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91"} Aug 05 18:19:18.348505 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.348502 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" event={"ID":"f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2","Type":"ContainerDied","Data":"9fb50b83e661326cdfd1d09719824819cbd4de3fd4f2c24d98614a0be87cb468"} Aug 05 18:19:18.348733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.348529 2561 scope.go:117] "RemoveContainer" containerID="b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91" Aug 05 18:19:18.348733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.348464 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8" Aug 05 18:19:18.357309 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.357295 2561 scope.go:117] "RemoveContainer" containerID="2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb" Aug 05 18:19:18.364855 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.364782 2561 scope.go:117] "RemoveContainer" containerID="b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91" Aug 05 18:19:18.365291 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:19:18.365265 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91\": container with ID starting with b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91 not found: ID does not exist" containerID="b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91" Aug 05 18:19:18.365404 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.365304 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91"} err="failed to get container status \"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91\": rpc error: code = NotFound desc = could not find container \"b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91\": container with ID starting with b0ab1a4251685572ab0c2c5c92f54de43d85ace22ea9a83b72802210aaa91e91 not found: ID does not exist" Aug 05 18:19:18.365404 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.365324 2561 scope.go:117] "RemoveContainer" containerID="2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb" Aug 05 18:19:18.365642 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:19:18.365614 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb\": container with ID starting with 2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb not found: ID does not exist" containerID="2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb" Aug 05 18:19:18.365714 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.365651 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb"} err="failed to get container status \"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb\": rpc error: code = NotFound desc = could not find container \"2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb\": container with ID starting with 2ced4bab9aeb06260273bf799bc93255e348341236fa03117d86a135efd275fb not found: ID does not exist" Aug 05 18:19:18.366903 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.366883 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:19:18.370768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:18.370747 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-acb45a-openshift-default-6f5c75bbd8-shfz8"] Aug 05 18:19:20.068984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:20.068952 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" path="/var/lib/kubelet/pods/f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2/volumes" Aug 05 18:19:30.503836 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:30.503797 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:19:35.419047 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:35.419017 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:19:35.419454 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:35.419255 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" containerID="cri-o://d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b" gracePeriod=30 Aug 05 18:19:36.260495 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260459 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:19:36.260752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260739 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:19:36.260752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260753 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:19:36.260870 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260760 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:19:36.260870 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260766 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:19:36.260870 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.260834 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:19:36.262761 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.262744 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.265317 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.265291 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-70b4a4-openshift-default-dockercfg-dw2kd\"" Aug 05 18:19:36.276286 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.276257 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:19:36.339030 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.338990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339030 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsfv\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339116 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339320 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339307 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.339567 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.339338 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.440672 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.440672 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsfv\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440940 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.440971 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441175 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441106 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441565 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441565 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.441565 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.441446 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.442036 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.442009 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.443280 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.443258 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.443512 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.443492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.452308 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.452285 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.452936 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.452917 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsfv\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv\") pod \"e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.572671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.572571 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:36.711101 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.711075 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:19:36.713081 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:19:36.713059 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17e22fd_2d7c_4063_a755_c53ff58685cc.slice/crio-9649891b48c7dee2c052cedb6e7b7801b4beea4958a284340a538a63d7b15392 WatchSource:0}: Error finding container 9649891b48c7dee2c052cedb6e7b7801b4beea4958a284340a538a63d7b15392: Status 404 returned error can't find the container with id 9649891b48c7dee2c052cedb6e7b7801b4beea4958a284340a538a63d7b15392 Aug 05 18:19:36.715100 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.715064 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:36.715174 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.715136 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:36.715174 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:36.715163 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:37.439898 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:37.439860 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerStarted","Data":"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6"} Aug 05 18:19:37.440103 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:37.439920 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerStarted","Data":"9649891b48c7dee2c052cedb6e7b7801b4beea4958a284340a538a63d7b15392"} Aug 05 18:19:37.463105 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:37.463050 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podStartSLOduration=1.463029155 podStartE2EDuration="1.463029155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:19:37.461349917 +0000 UTC m=+2803.937632396" watchObservedRunningTime="2026-08-05 18:19:37.463029155 +0000 UTC m=+2803.939311623" Aug 05 18:19:37.573391 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:37.573358 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:37.580430 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:37.580397 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:38.005171 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:38.005131 2561 patch_prober.go:28] interesting pod/e2e-api-b-87a594-openshift-default-69887dffbb-njts4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 05 18:19:38.005352 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:38.005188 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 05 18:19:38.443845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:38.443756 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:38.444872 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:38.444845 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:40.675678 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.675652 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:19:40.778455 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778405 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778623 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778486 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778623 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778516 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778623 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778576 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778623 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778603 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778654 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hps6w\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778678 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778712 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778734 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data\") pod \"a7c20020-8e62-4876-ad65-53c433578932\" (UID: \"a7c20020-8e62-4876-ad65-53c433578932\") " Aug 05 18:19:40.778850 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778745 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:40.779154 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.778992 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.779264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.779138 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:19:40.779332 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.779260 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:40.779332 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.779164 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:40.779459 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.779427 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:40.780877 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.780853 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:19:40.780957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.780945 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:19:40.781350 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.781332 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:19:40.781423 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.781370 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w" (OuterVolumeSpecName: "kube-api-access-hps6w") pod "a7c20020-8e62-4876-ad65-53c433578932" (UID: "a7c20020-8e62-4876-ad65-53c433578932"). InnerVolumeSpecName "kube-api-access-hps6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:19:40.880444 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880411 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880444 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880442 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880444 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880451 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880458 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880467 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hps6w\" (UniqueName: \"kubernetes.io/projected/a7c20020-8e62-4876-ad65-53c433578932-kube-api-access-hps6w\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880476 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c20020-8e62-4876-ad65-53c433578932-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880484 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c20020-8e62-4876-ad65-53c433578932-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:40.880679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:40.880491 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c20020-8e62-4876-ad65-53c433578932-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:19:41.460339 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.460306 2561 generic.go:356] "Generic (PLEG): container finished" podID="a7c20020-8e62-4876-ad65-53c433578932" containerID="d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b" exitCode=0 Aug 05 18:19:41.460545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.460348 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerDied","Data":"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b"} Aug 05 18:19:41.460545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.460360 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" Aug 05 18:19:41.460545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.460388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4" event={"ID":"a7c20020-8e62-4876-ad65-53c433578932","Type":"ContainerDied","Data":"c8ddcefc3b3552ab62c6ccfe048b2e531458c0fad21465b6581bd761c8d03475"} Aug 05 18:19:41.460545 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.460412 2561 scope.go:117] "RemoveContainer" containerID="d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b" Aug 05 18:19:41.469269 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.469251 2561 scope.go:117] "RemoveContainer" containerID="5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff" Aug 05 18:19:41.477338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.477317 2561 scope.go:117] "RemoveContainer" containerID="d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b" Aug 05 18:19:41.477604 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:19:41.477585 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b\": container with ID starting with d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b not found: ID does not exist" containerID="d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b" Aug 05 18:19:41.477657 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.477613 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b"} err="failed to get container status \"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b\": rpc error: code = NotFound desc = could not find container \"d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b\": container with ID starting with d24a13646022680828f62eb48d977308ebc935d71dedcf743db9feeca6cf471b not found: ID does not exist" Aug 05 18:19:41.477657 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.477631 2561 scope.go:117] "RemoveContainer" containerID="5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff" Aug 05 18:19:41.477871 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:19:41.477853 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff\": container with ID starting with 5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff not found: ID does not exist" containerID="5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff" Aug 05 18:19:41.477927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.477880 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff"} err="failed to get container status \"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff\": rpc error: code = NotFound desc = could not find container \"5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff\": container with ID starting with 5ad546e93a60afea6e9c3aa4b2e9046932c311f71416f6157a5cab22b87a6dff not found: ID does not exist" Aug 05 18:19:41.480919 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.480893 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:19:41.487310 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:41.487285 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-87a594-openshift-default-69887dffbb-njts4"] Aug 05 18:19:42.068588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:42.068555 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c20020-8e62-4876-ad65-53c433578932" path="/var/lib/kubelet/pods/a7c20020-8e62-4876-ad65-53c433578932/volumes" Aug 05 18:19:44.204556 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:44.204521 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:19:53.518661 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:53.518613 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": read tcp 10.132.0.2:41454->10.132.0.39:15021: read: connection reset by peer" start-of-body= Aug 05 18:19:53.519042 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:53.518706 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": read tcp 10.132.0.2:41454->10.132.0.39:15021: read: connection reset by peer" Aug 05 18:19:54.522564 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.522531 2561 generic.go:356] "Generic (PLEG): container finished" podID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerID="dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6" exitCode=0 Aug 05 18:19:54.523053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.522613 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerDied","Data":"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6"} Aug 05 18:19:54.523120 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.523107 2561 scope.go:117] "RemoveContainer" containerID="dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6" Aug 05 18:19:54.524079 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.524050 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:54.524183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.524133 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:54.524183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:54.524177 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:19:55.528357 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:55.528324 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerStarted","Data":"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e"} Aug 05 18:19:55.573553 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:55.573514 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:19:55.573794 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:55.573774 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:19:55.573854 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:55.573820 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Aug 05 18:19:56.573314 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:56.573276 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:19:56.573676 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:56.573356 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Aug 05 18:19:57.573864 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:57.573828 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:19:57.574275 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:57.573884 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Aug 05 18:19:58.573971 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:58.573934 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:19:58.574408 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:58.573991 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Aug 05 18:19:59.573471 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:59.573437 2561 patch_prober.go:28] interesting pod/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Aug 05 18:19:59.573658 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:19:59.573489 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Aug 05 18:20:00.577455 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:00.577427 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:20:00.577902 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:00.577774 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:20:00.578564 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:00.578543 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:20:05.974315 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974283 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974597 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974610 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974618 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974625 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974671 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0dd1708-d2f1-489c-ae04-ec3e1df8b6f2" containerName="istio-proxy" Aug 05 18:20:05.974679 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974678 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.974868 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.974686 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c20020-8e62-4876-ad65-53c433578932" containerName="istio-proxy" Aug 05 18:20:05.976541 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.976525 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.979084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.979063 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-6fb560-openshift-default-dockercfg-zbsg4\"" Aug 05 18:20:05.991525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.991502 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:20:05.993712 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993692 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993800 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993759 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993887 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993822 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993887 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993965 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.993965 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.994074 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.993975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:05.994074 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:05.994019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4446\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095047 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095013 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095273 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4446\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095273 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095273 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095273 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095285 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095313 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095340 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095457 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095771 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095566 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095771 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095581 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.095771 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.095684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.096116 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.096092 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.098072 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.098045 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.098164 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.098100 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.103557 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.103533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.103700 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.103680 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4446\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446\") pod \"e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.286144 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.286112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:06.421725 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.421701 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:20:06.423864 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:20:06.423835 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e59029_a89f_46df_8a54_58478a07b0f7.slice/crio-9a627bedada211b7c0d48c73b0040adf6218f5b9116302fae279a8c2ff7ebe20 WatchSource:0}: Error finding container 9a627bedada211b7c0d48c73b0040adf6218f5b9116302fae279a8c2ff7ebe20: Status 404 returned error can't find the container with id 9a627bedada211b7c0d48c73b0040adf6218f5b9116302fae279a8c2ff7ebe20 Aug 05 18:20:06.426159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.426121 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:06.426298 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.426222 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:06.426298 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.426277 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:06.580925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.580819 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerStarted","Data":"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e"} Aug 05 18:20:06.580925 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.580888 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerStarted","Data":"9a627bedada211b7c0d48c73b0040adf6218f5b9116302fae279a8c2ff7ebe20"} Aug 05 18:20:06.603114 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:06.603060 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podStartSLOduration=1.603044581 podStartE2EDuration="1.603044581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:06.601746605 +0000 UTC m=+2833.078029071" watchObservedRunningTime="2026-08-05 18:20:06.603044581 +0000 UTC m=+2833.079327045" Aug 05 18:20:07.287105 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:07.287067 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:07.292358 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:07.292324 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:07.585297 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:07.585192 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:07.586157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:07.586139 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:12.600239 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:12.600192 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:20:22.586631 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.586589 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:22.587079 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.586665 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:22.653288 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.653254 2561 generic.go:356] "Generic (PLEG): container finished" podID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerID="88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e" exitCode=0 Aug 05 18:20:22.653501 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.653329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerDied","Data":"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e"} Aug 05 18:20:22.653781 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.653763 2561 scope.go:117] "RemoveContainer" containerID="88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e" Aug 05 18:20:22.654827 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.654799 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:22.654889 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.654865 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:22.654927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:22.654895 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:20:23.659132 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:23.659097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerStarted","Data":"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71"} Aug 05 18:20:24.287031 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:24.286988 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:24.287469 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:24.287436 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:24.287602 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:24.287496 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:25.287588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:25.287541 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:25.288111 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:25.287628 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:26.286964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:26.286916 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:26.287194 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:26.286998 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:27.286857 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:27.286805 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:27.287408 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:27.286880 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:28.300759 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:28.300715 2561 patch_prober.go:28] interesting pod/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Aug 05 18:20:28.301516 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:28.301480 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Aug 05 18:20:29.291671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:29.291634 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:29.292138 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:29.292109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:29.292884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:29.292862 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:20:39.231849 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.231806 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:20:39.234078 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.234055 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.236653 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.236622 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"kube-root-ca.crt\"" Aug 05 18:20:39.237620 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.237601 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"auth-test-model-70b4a4-kserve-self-signed-certs\"" Aug 05 18:20:39.237620 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.237617 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"openshift-service-ca.crt\"" Aug 05 18:20:39.237620 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.237628 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"default-dockercfg-ksdpv\"" Aug 05 18:20:39.246095 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.246073 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:20:39.397740 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.397703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.397957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.397749 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpvg\" (UniqueName: \"kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.397957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.397773 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.397957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.397882 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.397957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.397925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.398116 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.398018 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.498904 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.498875 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.498924 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.498965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpvg\" (UniqueName: \"kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.498996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.499039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499084 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.499063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499441 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.499415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499550 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.499440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.499550 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.499495 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.501565 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.501543 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.501727 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.501711 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.507372 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.507349 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpvg\" (UniqueName: \"kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg\") pod \"auth-test-model-70b4a4-kserve-584f86f96c-w9czr\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.543413 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.543372 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:39.671153 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.671129 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:20:39.673779 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:20:39.673753 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcef5e72a_3154_4a85_ba02_d7ebc507b71e.slice/crio-3b5649dbd52254e2e68e2b2ce6640d9e9e5ba2fe2ef763632cb5c820447faccd WatchSource:0}: Error finding container 3b5649dbd52254e2e68e2b2ce6640d9e9e5ba2fe2ef763632cb5c820447faccd: Status 404 returned error can't find the container with id 3b5649dbd52254e2e68e2b2ce6640d9e9e5ba2fe2ef763632cb5c820447faccd Aug 05 18:20:39.725306 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:39.725273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" event={"ID":"cef5e72a-3154-4a85-ba02-d7ebc507b71e","Type":"ContainerStarted","Data":"3b5649dbd52254e2e68e2b2ce6640d9e9e5ba2fe2ef763632cb5c820447faccd"} Aug 05 18:20:40.730530 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:40.730492 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" event={"ID":"cef5e72a-3154-4a85-ba02-d7ebc507b71e","Type":"ContainerStarted","Data":"cd8fcbc895bafa5c3d197a08f3ad802a10394296ef3600eddd22569217d89191"} Aug 05 18:20:40.750500 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:40.750448 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" podStartSLOduration=1.7504299749999999 podStartE2EDuration="1.750429975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:40.747932235 +0000 UTC m=+2867.224214699" watchObservedRunningTime="2026-08-05 18:20:40.750429975 +0000 UTC m=+2867.226712441" Aug 05 18:20:41.708557 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:41.708520 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:20:49.543873 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:49.543789 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:49.543873 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:49.543841 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:49.556462 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:49.556431 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:20:49.781402 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:20:49.781368 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:21:01.317113 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:21:01.317080 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:21:23.404804 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:21:23.404757 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:21:37.099446 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:21:37.099345 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:21:48.097313 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:21:48.097283 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:22:03.415957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:03.415919 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:22:14.306162 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:14.306130 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:22:29.734256 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:29.734203 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:22:54.151627 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:54.151597 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:22:54.153984 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:54.153950 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:22:55.405995 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:22:55.405963 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:23:06.203968 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:23:06.203930 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:23:22.112474 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:23:22.112438 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:23:37.400159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:23:37.400111 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:23:48.718278 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:23:48.718176 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:24:03.900986 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:03.900948 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:24:14.610323 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:14.610288 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:24:30.804918 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:30.804884 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:24:34.604607 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:34.604578 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:24:41.601875 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:41.601838 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:24:49.623576 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.623510 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerName="main" containerID="cri-o://cd8fcbc895bafa5c3d197a08f3ad802a10394296ef3600eddd22569217d89191" gracePeriod=45 Aug 05 18:24:49.771126 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.771089 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": dial tcp 10.132.0.41:8000: connect: connection refused" Aug 05 18:24:49.777563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.777529 2561 generic.go:356] "Generic (PLEG): container finished" podID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerID="cd8fcbc895bafa5c3d197a08f3ad802a10394296ef3600eddd22569217d89191" exitCode=0 Aug 05 18:24:49.777691 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.777606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" event={"ID":"cef5e72a-3154-4a85-ba02-d7ebc507b71e","Type":"ContainerDied","Data":"cd8fcbc895bafa5c3d197a08f3ad802a10394296ef3600eddd22569217d89191"} Aug 05 18:24:49.865992 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.865966 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:24:49.987476 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987390 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987476 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987425 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987476 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987459 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987483 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987524 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zpvg\" (UniqueName: \"kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987576 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home\") pod \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\" (UID: \"cef5e72a-3154-4a85-ba02-d7ebc507b71e\") " Aug 05 18:24:49.987903 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987758 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache" (OuterVolumeSpecName: "model-cache") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:24:49.987903 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987786 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:24:49.987903 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.987859 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home" (OuterVolumeSpecName: "home") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:24:49.989806 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.989775 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm" (OuterVolumeSpecName: "dshm") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:24:49.989927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.989862 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:24:49.989927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:49.989871 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg" (OuterVolumeSpecName: "kube-api-access-9zpvg") pod "cef5e72a-3154-4a85-ba02-d7ebc507b71e" (UID: "cef5e72a-3154-4a85-ba02-d7ebc507b71e"). InnerVolumeSpecName "kube-api-access-9zpvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:24:50.088467 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088436 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.088467 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088463 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.088467 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088473 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.088671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088497 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.088671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088506 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpvg\" (UniqueName: \"kubernetes.io/projected/cef5e72a-3154-4a85-ba02-d7ebc507b71e-kube-api-access-9zpvg\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.088671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.088514 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef5e72a-3154-4a85-ba02-d7ebc507b71e-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:24:50.783210 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.783180 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" Aug 05 18:24:50.783210 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.783188 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr" event={"ID":"cef5e72a-3154-4a85-ba02-d7ebc507b71e","Type":"ContainerDied","Data":"3b5649dbd52254e2e68e2b2ce6640d9e9e5ba2fe2ef763632cb5c820447faccd"} Aug 05 18:24:50.783768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.783254 2561 scope.go:117] "RemoveContainer" containerID="cd8fcbc895bafa5c3d197a08f3ad802a10394296ef3600eddd22569217d89191" Aug 05 18:24:50.800277 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.800249 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:24:50.803708 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:50.803684 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/auth-test-model-70b4a4-kserve-584f86f96c-w9czr"] Aug 05 18:24:52.069176 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:52.069132 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" path="/var/lib/kubelet/pods/cef5e72a-3154-4a85-ba02-d7ebc507b71e/volumes" Aug 05 18:24:57.047463 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.047429 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:24:57.047845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.047737 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerName="main" Aug 05 18:24:57.047845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.047752 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerName="main" Aug 05 18:24:57.047845 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.047824 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef5e72a-3154-4a85-ba02-d7ebc507b71e" containerName="main" Aug 05 18:24:57.050606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.050587 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.054166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.054141 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"kube-root-ca.crt\"" Aug 05 18:24:57.054166 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.054154 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"openshift-service-ca.crt\"" Aug 05 18:24:57.054407 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.054169 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"sub-test-model-70b4a4-kserve-self-signed-certs\"" Aug 05 18:24:57.054471 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.054438 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"default-dockercfg-ksdpv\"" Aug 05 18:24:57.059544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.059519 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:24:57.150843 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.150810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.151039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.150858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.151039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.150874 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.151039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.150927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.151039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.150958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q562m\" (UniqueName: \"kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.151039 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.151028 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251456 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251456 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251617 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q562m\" (UniqueName: \"kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251718 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251888 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251842 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.251957 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251926 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.252024 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.251989 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.252191 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.252143 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.253938 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.253917 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.254215 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.254195 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.260907 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.260888 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q562m\" (UniqueName: \"kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m\") pod \"sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.360392 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.360282 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:24:57.493904 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.493878 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:24:57.495657 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:24:57.495630 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ce92ac_dcda_4fff_bf6a_1f825819abe6.slice/crio-ae10cfc6c8d37e9e516bd46e63a09aebc05c79f275c5500cbb4d8622ccc9a3ce WatchSource:0}: Error finding container ae10cfc6c8d37e9e516bd46e63a09aebc05c79f275c5500cbb4d8622ccc9a3ce: Status 404 returned error can't find the container with id ae10cfc6c8d37e9e516bd46e63a09aebc05c79f275c5500cbb4d8622ccc9a3ce Aug 05 18:24:57.815706 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.815670 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" event={"ID":"62ce92ac-dcda-4fff-bf6a-1f825819abe6","Type":"ContainerStarted","Data":"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c"} Aug 05 18:24:57.815878 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.815716 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" event={"ID":"62ce92ac-dcda-4fff-bf6a-1f825819abe6","Type":"ContainerStarted","Data":"ae10cfc6c8d37e9e516bd46e63a09aebc05c79f275c5500cbb4d8622ccc9a3ce"} Aug 05 18:24:57.835585 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:57.835537 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" podStartSLOduration=0.835521655 podStartE2EDuration="835.521655ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:24:57.834788022 +0000 UTC m=+3124.311070488" watchObservedRunningTime="2026-08-05 18:24:57.835521655 +0000 UTC m=+3124.311804125" Aug 05 18:24:59.802158 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:24:59.802123 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:25:07.361308 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:25:07.361271 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:25:07.361730 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:25:07.361324 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:25:07.373729 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:25:07.373701 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:25:07.873862 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:25:07.873823 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:25:26.196575 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:25:26.196541 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:08.211567 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:08.211531 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:21.103706 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:21.103663 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:32.603768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:32.603728 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:38.308697 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:38.308665 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:26:42.223894 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:42.223855 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:47.339277 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.339179 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:26:47.342243 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.342212 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.344915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.344894 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-70b4a4\"/\"rate-test-model-70b4a4-kserve-self-signed-certs\"" Aug 05 18:26:47.352581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.352556 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:26:47.390021 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.389976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.390158 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.390038 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.390158 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.390069 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.390313 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.390162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.390313 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.390213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.390313 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.390277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4z97\" (UniqueName: \"kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.490747 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490706 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.490923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490764 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.490923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490810 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.490923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490876 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.490923 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.491136 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.490936 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4z97\" (UniqueName: \"kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.491199 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.491129 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.491288 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.491263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.491354 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.491319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.493520 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.493491 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.493520 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.493504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.500184 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.500125 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4z97\" (UniqueName: \"kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97\") pod \"rate-test-model-70b4a4-kserve-d79f58759-9xtkr\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.652100 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.652017 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:47.781952 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:47.781927 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:26:47.784059 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:26:47.784025 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08029f05_d94d_42c1_8d30_0682c27c158c.slice/crio-20fda28b95b9be2053c062b8784f83484fabed52893a02ab25783fc660e83784 WatchSource:0}: Error finding container 20fda28b95b9be2053c062b8784f83484fabed52893a02ab25783fc660e83784: Status 404 returned error can't find the container with id 20fda28b95b9be2053c062b8784f83484fabed52893a02ab25783fc660e83784 Aug 05 18:26:48.299210 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:48.299178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" event={"ID":"08029f05-d94d-42c1-8d30-0682c27c158c","Type":"ContainerStarted","Data":"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a"} Aug 05 18:26:48.299376 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:48.299218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" event={"ID":"08029f05-d94d-42c1-8d30-0682c27c158c","Type":"ContainerStarted","Data":"20fda28b95b9be2053c062b8784f83484fabed52893a02ab25783fc660e83784"} Aug 05 18:26:48.325296 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:48.325245 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" podStartSLOduration=1.3252130100000001 podStartE2EDuration="1.32521301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:26:48.324370751 +0000 UTC m=+3234.800653410" watchObservedRunningTime="2026-08-05 18:26:48.32521301 +0000 UTC m=+3234.801495452" Aug 05 18:26:49.207787 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:49.207750 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:26:53.327586 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.327526 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" podUID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" containerName="main" containerID="cri-o://6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c" gracePeriod=45 Aug 05 18:26:53.570593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.570567 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:26:53.643525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643442 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q562m\" (UniqueName: \"kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643489 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643525 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643861 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643543 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643861 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643567 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643861 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643588 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home\") pod \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\" (UID: \"62ce92ac-dcda-4fff-bf6a-1f825819abe6\") " Aug 05 18:26:53.643861 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643804 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:26:53.644042 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643940 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache" (OuterVolumeSpecName: "model-cache") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:26:53.644042 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.643961 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home" (OuterVolumeSpecName: "home") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:26:53.645846 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.645823 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:26:53.645846 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.645840 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm" (OuterVolumeSpecName: "dshm") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:26:53.646029 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.645851 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m" (OuterVolumeSpecName: "kube-api-access-q562m") pod "62ce92ac-dcda-4fff-bf6a-1f825819abe6" (UID: "62ce92ac-dcda-4fff-bf6a-1f825819abe6"). InnerVolumeSpecName "kube-api-access-q562m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:26:53.744511 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744469 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:53.744511 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744501 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:53.744511 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744510 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:53.744511 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744518 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q562m\" (UniqueName: \"kubernetes.io/projected/62ce92ac-dcda-4fff-bf6a-1f825819abe6-kube-api-access-q562m\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:53.744789 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744528 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:53.744789 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:53.744536 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ce92ac-dcda-4fff-bf6a-1f825819abe6-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:26:54.325485 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.325448 2561 generic.go:356] "Generic (PLEG): container finished" podID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" containerID="6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c" exitCode=0 Aug 05 18:26:54.325662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.325511 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" Aug 05 18:26:54.325662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.325536 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" event={"ID":"62ce92ac-dcda-4fff-bf6a-1f825819abe6","Type":"ContainerDied","Data":"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c"} Aug 05 18:26:54.325662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.325583 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq" event={"ID":"62ce92ac-dcda-4fff-bf6a-1f825819abe6","Type":"ContainerDied","Data":"ae10cfc6c8d37e9e516bd46e63a09aebc05c79f275c5500cbb4d8622ccc9a3ce"} Aug 05 18:26:54.325662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.325606 2561 scope.go:117] "RemoveContainer" containerID="6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c" Aug 05 18:26:54.334183 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.333945 2561 scope.go:117] "RemoveContainer" containerID="6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c" Aug 05 18:26:54.334435 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:26:54.334203 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c\": container with ID starting with 6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c not found: ID does not exist" containerID="6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c" Aug 05 18:26:54.334435 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.334241 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c"} err="failed to get container status \"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c\": rpc error: code = NotFound desc = could not find container \"6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c\": container with ID starting with 6a56d7682113c2e0784529ee7f861e86f7bb2bb98b870d2b0d0bd4765557222c not found: ID does not exist" Aug 05 18:26:54.342604 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.342580 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:26:54.346651 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:54.346630 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/sub-test-model-70b4a4-kserve-6bcc7545c5-p85mq"] Aug 05 18:26:56.068323 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:56.068289 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" path="/var/lib/kubelet/pods/62ce92ac-dcda-4fff-bf6a-1f825819abe6/volumes" Aug 05 18:26:57.652725 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:57.652688 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:57.653194 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:57.652748 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:57.665640 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:57.665610 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:26:58.356131 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:26:58.356099 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:27:09.808879 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:27:09.808846 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:27:54.175262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:27:54.175219 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:27:54.178323 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:27:54.178300 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:27:56.006445 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:27:56.006406 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:28:03.905445 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:03.905409 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:28:20.405285 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:20.405201 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:28:25.747616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:25.747585 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:28:28.497033 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:28.497001 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:28:40.766377 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:40.766339 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" podUID="08029f05-d94d-42c1-8d30-0682c27c158c" containerName="main" containerID="cri-o://b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a" gracePeriod=45 Aug 05 18:28:41.016463 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.016393 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:28:41.061769 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.061738 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.061908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.061783 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.061908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.061805 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.061908 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.061821 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.062081 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.061941 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.062081 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062026 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4z97\" (UniqueName: \"kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97\") pod \"08029f05-d94d-42c1-8d30-0682c27c158c\" (UID: \"08029f05-d94d-42c1-8d30-0682c27c158c\") " Aug 05 18:28:41.062081 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062052 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home" (OuterVolumeSpecName: "home") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:28:41.062081 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062063 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:28:41.062308 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062176 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache" (OuterVolumeSpecName: "model-cache") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:28:41.062366 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062319 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-home\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.062366 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062340 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-tmp-dir\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.062366 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.062354 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-model-cache\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.063917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.063892 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 05 18:28:41.064063 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.063982 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm" (OuterVolumeSpecName: "dshm") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:28:41.064063 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.064040 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97" (OuterVolumeSpecName: "kube-api-access-m4z97") pod "08029f05-d94d-42c1-8d30-0682c27c158c" (UID: "08029f05-d94d-42c1-8d30-0682c27c158c"). InnerVolumeSpecName "kube-api-access-m4z97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:28:41.163213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.163189 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08029f05-d94d-42c1-8d30-0682c27c158c-tls-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.163213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.163212 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08029f05-d94d-42c1-8d30-0682c27c158c-dshm\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.163383 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.163221 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4z97\" (UniqueName: \"kubernetes.io/projected/08029f05-d94d-42c1-8d30-0682c27c158c-kube-api-access-m4z97\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:28:41.773451 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.773415 2561 generic.go:356] "Generic (PLEG): container finished" podID="08029f05-d94d-42c1-8d30-0682c27c158c" containerID="b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a" exitCode=0 Aug 05 18:28:41.773924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.773450 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" event={"ID":"08029f05-d94d-42c1-8d30-0682c27c158c","Type":"ContainerDied","Data":"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a"} Aug 05 18:28:41.773924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.773482 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" Aug 05 18:28:41.773924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.773494 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr" event={"ID":"08029f05-d94d-42c1-8d30-0682c27c158c","Type":"ContainerDied","Data":"20fda28b95b9be2053c062b8784f83484fabed52893a02ab25783fc660e83784"} Aug 05 18:28:41.773924 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.773513 2561 scope.go:117] "RemoveContainer" containerID="b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a" Aug 05 18:28:41.782314 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.782298 2561 scope.go:117] "RemoveContainer" containerID="b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a" Aug 05 18:28:41.782557 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:28:41.782540 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a\": container with ID starting with b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a not found: ID does not exist" containerID="b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a" Aug 05 18:28:41.782606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.782565 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a"} err="failed to get container status \"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a\": rpc error: code = NotFound desc = could not find container \"b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a\": container with ID starting with b340b2d82a7fcb22e556ed58e458cf75f61661020973e00d740e0defd3a1031a not found: ID does not exist" Aug 05 18:28:41.796353 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.796328 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:28:41.800915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:41.800893 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-70b4a4/rate-test-model-70b4a4-kserve-d79f58759-9xtkr"] Aug 05 18:28:42.069264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:42.069151 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08029f05-d94d-42c1-8d30-0682c27c158c" path="/var/lib/kubelet/pods/08029f05-d94d-42c1-8d30-0682c27c158c/volumes" Aug 05 18:28:46.042017 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.041978 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042330 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08029f05-d94d-42c1-8d30-0682c27c158c" containerName="main" Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042343 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08029f05-d94d-42c1-8d30-0682c27c158c" containerName="main" Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042354 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" containerName="main" Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042359 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" containerName="main" Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042407 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ce92ac-dcda-4fff-bf6a-1f825819abe6" containerName="main" Aug 05 18:28:46.042412 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.042416 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08029f05-d94d-42c1-8d30-0682c27c158c" containerName="main" Aug 05 18:28:46.046604 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.046587 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.049108 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.049086 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b5b093-openshift-default-dockercfg-kg7dc\"" Aug 05 18:28:46.058531 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.058503 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:28:46.102882 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.102842 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.102882 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.102889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.102916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.102942 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnc9\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.102964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.103001 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.103040 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.103058 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.103109 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.103093 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.203954 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.203904 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.203954 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.203959 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.203984 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204022 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204107 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204153 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204264 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204186 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnc9\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204644 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204644 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204489 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204755 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204646 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204811 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204829 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.204927 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.204885 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.206575 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.206535 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.206723 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.206702 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.213371 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.213352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.214037 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.214015 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnc9\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9\") pod \"e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.357963 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.357873 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:46.496813 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.496787 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:28:46.499263 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:28:46.499218 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc642a85e_c872_4bff_9c90_8eb36eb52423.slice/crio-63a2662452dc25bcbb4a4bcdf48e2c905367e0b6a0905dbaf5976d0e69a00a68 WatchSource:0}: Error finding container 63a2662452dc25bcbb4a4bcdf48e2c905367e0b6a0905dbaf5976d0e69a00a68: Status 404 returned error can't find the container with id 63a2662452dc25bcbb4a4bcdf48e2c905367e0b6a0905dbaf5976d0e69a00a68 Aug 05 18:28:46.501356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.501323 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:28:46.501421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.501388 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:28:46.501421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.501417 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:28:46.796741 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.796701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerStarted","Data":"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235"} Aug 05 18:28:46.796916 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.796765 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerStarted","Data":"63a2662452dc25bcbb4a4bcdf48e2c905367e0b6a0905dbaf5976d0e69a00a68"} Aug 05 18:28:46.819439 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:46.819396 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podStartSLOduration=0.819379968 podStartE2EDuration="819.379968ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:28:46.817342409 +0000 UTC m=+3353.293624877" watchObservedRunningTime="2026-08-05 18:28:46.819379968 +0000 UTC m=+3353.295662432" Aug 05 18:28:47.358033 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:47.358003 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:47.363012 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:47.362981 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:47.800662 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:47.800625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:47.801686 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:47.801666 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:28:53.125753 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:28:53.125713 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:29:03.077387 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.076896 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": read tcp 10.132.0.2:48828->10.132.0.44:15021: read: connection reset by peer" start-of-body= Aug 05 18:29:03.077387 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.076958 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": read tcp 10.132.0.2:48828->10.132.0.44:15021: read: connection reset by peer" Aug 05 18:29:03.874344 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.874311 2561 generic.go:356] "Generic (PLEG): container finished" podID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerID="79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235" exitCode=0 Aug 05 18:29:03.874524 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.874385 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerDied","Data":"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235"} Aug 05 18:29:03.874752 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.874739 2561 scope.go:117] "RemoveContainer" containerID="79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235" Aug 05 18:29:03.875761 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.875731 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:03.875865 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.875832 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:03.875929 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:03.875884 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:04.881211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:04.881171 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerStarted","Data":"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc"} Aug 05 18:29:05.358262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:05.358218 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:29:05.358475 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:05.358450 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:05.358530 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:05.358509 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:06.358974 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:06.358939 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:06.359389 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:06.359007 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:07.358783 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:07.358742 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:07.358965 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:07.358803 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:08.358337 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:08.358298 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:08.358734 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:08.358367 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:09.359034 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:09.358997 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:09.359454 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:09.359060 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:10.359169 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:10.359132 2561 patch_prober.go:28] interesting pod/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Aug 05 18:29:10.359592 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:10.359187 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Aug 05 18:29:11.362513 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:11.362481 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:29:11.362921 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:11.362751 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:29:11.363897 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:11.363878 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:29:14.636768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.636733 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:29:14.640246 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.640211 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.642844 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.642821 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cea6f9-openshift-default-dockercfg-pc6js\"" Aug 05 18:29:14.654817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.654793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:29:14.760323 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760294 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760497 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760349 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760497 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760387 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760497 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8fb\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760497 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760545 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760586 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.760647 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.760631 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.861982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.861946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.861982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.861987 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8fb\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862262 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862484 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862484 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862365 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862484 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862377 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862553 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862648 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862631 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862698 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862634 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.862783 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.862763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.863143 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.863125 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.864569 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.864539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.864816 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.864796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.870110 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.870082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.870588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.870568 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8fb\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb\") pod \"e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:14.950142 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:14.950051 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:15.087527 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.087499 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:29:15.089702 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:29:15.089674 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0458cf14_88a1_4bd9_b887_360ff6b3ecc2.slice/crio-e5651d8d080a24a427e0b8c0ccd8f9f48cf2f863b19c61527ce4421bc21003f4 WatchSource:0}: Error finding container e5651d8d080a24a427e0b8c0ccd8f9f48cf2f863b19c61527ce4421bc21003f4: Status 404 returned error can't find the container with id e5651d8d080a24a427e0b8c0ccd8f9f48cf2f863b19c61527ce4421bc21003f4 Aug 05 18:29:15.091915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.091877 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:15.091982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.091969 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:15.092020 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.092008 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:15.936921 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.936885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerStarted","Data":"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75"} Aug 05 18:29:15.937367 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.936935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerStarted","Data":"e5651d8d080a24a427e0b8c0ccd8f9f48cf2f863b19c61527ce4421bc21003f4"} Aug 05 18:29:15.950290 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.950269 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:15.959671 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:15.959628 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podStartSLOduration=1.959614293 podStartE2EDuration="1.959614293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:29:15.957311587 +0000 UTC m=+3382.433594052" watchObservedRunningTime="2026-08-05 18:29:15.959614293 +0000 UTC m=+3382.435896758" Aug 05 18:29:16.955185 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:16.955153 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:17.946540 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:17.946514 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:17.947521 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:17.947502 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:22.408158 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:22.408127 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:29:32.947988 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:32.947948 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:32.948430 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:32.948019 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:33.023390 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.023353 2561 generic.go:356] "Generic (PLEG): container finished" podID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerID="5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75" exitCode=0 Aug 05 18:29:33.023390 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.023394 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerDied","Data":"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75"} Aug 05 18:29:33.023749 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.023735 2561 scope.go:117] "RemoveContainer" containerID="5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75" Aug 05 18:29:33.024873 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.024837 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:33.024946 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.024907 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:33.024946 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:33.024937 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Aug 05 18:29:34.030219 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:34.030183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerStarted","Data":"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08"} Aug 05 18:29:34.950799 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:34.950769 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:34.950979 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:34.950944 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:34.951408 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:34.951365 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:35.950625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:35.950588 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:35.951009 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:35.950648 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:36.950669 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:36.950631 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:36.951065 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:36.950710 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:37.951080 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:37.951037 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:37.951489 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:37.951115 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:38.951217 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:38.951178 2561 patch_prober.go:28] interesting pod/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Aug 05 18:29:38.951661 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:38.951267 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Aug 05 18:29:39.954741 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:39.954712 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:39.955149 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:39.954975 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:29:39.955594 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:29:39.955580 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:30:01.509990 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:01.509947 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:30:04.724808 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:04.724771 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd"] Aug 05 18:30:04.734816 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:04.734789 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29765865-zswgd"] Aug 05 18:30:06.068725 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:06.068691 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c91b06-c868-4169-8a1d-b609937aee57" path="/var/lib/kubelet/pods/97c91b06-c868-4169-8a1d-b609937aee57/volumes" Aug 05 18:30:23.505040 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:23.504992 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:30:46.107121 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:46.107075 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:30:54.329452 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:30:54.329412 2561 scope.go:117] "RemoveContainer" containerID="d695ac5caaba11cbd18ecf932ab8c576e3c84b7e9f66d92b13cfa3a39a0a2f67" Aug 05 18:31:04.456907 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.454565 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l"] Aug 05 18:31:04.460684 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.460648 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:04.464246 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.464200 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:31:04.464246 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.464220 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 18:31:04.465667 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.465640 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l"] Aug 05 18:31:04.607901 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.607868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l\" (UID: \"6a051eac-318a-492e-a170-8797c382261f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:04.708623 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.708529 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l\" (UID: \"6a051eac-318a-492e-a170-8797c382261f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:04.709149 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.709130 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l\" (UID: \"6a051eac-318a-492e-a170-8797c382261f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:04.771993 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:04.771947 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:05.110195 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:05.110082 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l"] Aug 05 18:31:05.112664 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:31:05.112628 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a051eac_318a_492e_a170_8797c382261f.slice/crio-909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b WatchSource:0}: Error finding container 909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b: Status 404 returned error can't find the container with id 909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b Aug 05 18:31:05.426128 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:05.426028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" event={"ID":"6a051eac-318a-492e-a170-8797c382261f","Type":"ContainerStarted","Data":"871bba5ae4074bd3825de445a9ea0d9851f925e00be5276b19a79d399a99de36"} Aug 05 18:31:05.426128 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:05.426076 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" event={"ID":"6a051eac-318a-492e-a170-8797c382261f","Type":"ContainerStarted","Data":"909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b"} Aug 05 18:31:05.443828 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:05.443779 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" podStartSLOduration=1.443762919 podStartE2EDuration="1.443762919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:31:05.441687961 +0000 UTC m=+3491.917970428" watchObservedRunningTime="2026-08-05 18:31:05.443762919 +0000 UTC m=+3491.920045383" Aug 05 18:31:06.431986 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:06.431953 2561 generic.go:356] "Generic (PLEG): container finished" podID="6a051eac-318a-492e-a170-8797c382261f" containerID="871bba5ae4074bd3825de445a9ea0d9851f925e00be5276b19a79d399a99de36" exitCode=0 Aug 05 18:31:06.432421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:06.432014 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" event={"ID":"6a051eac-318a-492e-a170-8797c382261f","Type":"ContainerDied","Data":"871bba5ae4074bd3825de445a9ea0d9851f925e00be5276b19a79d399a99de36"} Aug 05 18:31:07.565385 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:07.565361 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:07.631354 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:07.631314 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca\") pod \"6a051eac-318a-492e-a170-8797c382261f\" (UID: \"6a051eac-318a-492e-a170-8797c382261f\") " Aug 05 18:31:07.631692 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:07.631667 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a051eac-318a-492e-a170-8797c382261f" (UID: "6a051eac-318a-492e-a170-8797c382261f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:31:07.732982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:07.732894 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a051eac-318a-492e-a170-8797c382261f-maas-api-service-ca\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:31:08.442762 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:08.442673 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" Aug 05 18:31:08.442762 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:08.442686 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l" event={"ID":"6a051eac-318a-492e-a170-8797c382261f","Type":"ContainerDied","Data":"909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b"} Aug 05 18:31:08.442762 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:08.442734 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909e3ccf9e9dd3972f5110b8b9cce501b2d43623a75bc27df18dba331d85eb1b" Aug 05 18:31:10.728521 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:10.728484 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l"] Aug 05 18:31:10.733322 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:10.733290 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eef9c6-8dk4l"] Aug 05 18:31:12.069106 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:12.069071 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a051eac-318a-492e-a170-8797c382261f" path="/var/lib/kubelet/pods/6a051eac-318a-492e-a170-8797c382261f/volumes" Aug 05 18:31:13.805798 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:13.805761 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:31:21.063090 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.063007 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs"] Aug 05 18:31:21.063492 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.063401 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a051eac-318a-492e-a170-8797c382261f" containerName="revoke-keys" Aug 05 18:31:21.063492 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.063415 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a051eac-318a-492e-a170-8797c382261f" containerName="revoke-keys" Aug 05 18:31:21.063492 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.063466 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a051eac-318a-492e-a170-8797c382261f" containerName="revoke-keys" Aug 05 18:31:21.066665 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.066646 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:21.069470 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.069442 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 05 18:31:21.070187 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.070169 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xn2vw\"" Aug 05 18:31:21.079696 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.079664 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs"] Aug 05 18:31:21.153733 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.153696 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs\" (UID: \"d84e6767-2c5f-49dc-a20b-92b1a43397a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:21.254458 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.254411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs\" (UID: \"d84e6767-2c5f-49dc-a20b-92b1a43397a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:21.255044 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.255022 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs\" (UID: \"d84e6767-2c5f-49dc-a20b-92b1a43397a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:21.376503 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.376407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:21.521973 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:21.521892 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs"] Aug 05 18:31:21.524810 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:31:21.524775 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84e6767_2c5f_49dc_a20b_92b1a43397a2.slice/crio-7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2 WatchSource:0}: Error finding container 7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2: Status 404 returned error can't find the container with id 7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2 Aug 05 18:31:22.508061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:22.508020 2561 generic.go:356] "Generic (PLEG): container finished" podID="d84e6767-2c5f-49dc-a20b-92b1a43397a2" containerID="515b349bd218d1385dafe3f94bf101bc5cb064078080b398e772dd11bcbaacb7" exitCode=0 Aug 05 18:31:22.508543 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:22.508071 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" event={"ID":"d84e6767-2c5f-49dc-a20b-92b1a43397a2","Type":"ContainerDied","Data":"515b349bd218d1385dafe3f94bf101bc5cb064078080b398e772dd11bcbaacb7"} Aug 05 18:31:22.508543 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:22.508115 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" event={"ID":"d84e6767-2c5f-49dc-a20b-92b1a43397a2","Type":"ContainerStarted","Data":"7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2"} Aug 05 18:31:23.648721 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:23.648690 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:23.777297 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:23.777170 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca\") pod \"d84e6767-2c5f-49dc-a20b-92b1a43397a2\" (UID: \"d84e6767-2c5f-49dc-a20b-92b1a43397a2\") " Aug 05 18:31:23.777572 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:23.777547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d84e6767-2c5f-49dc-a20b-92b1a43397a2" (UID: "d84e6767-2c5f-49dc-a20b-92b1a43397a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:31:23.877887 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:23.877835 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84e6767-2c5f-49dc-a20b-92b1a43397a2-maas-api-service-ca\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:31:24.518708 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:24.518673 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" event={"ID":"d84e6767-2c5f-49dc-a20b-92b1a43397a2","Type":"ContainerDied","Data":"7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2"} Aug 05 18:31:24.518708 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:24.518705 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs" Aug 05 18:31:24.518939 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:24.518727 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d8ef0993785ae26fb37bbd63462cf067e7f6463f300d5ddceb33b85e574f5e2" Aug 05 18:31:34.912403 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:34.912365 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:31:36.818724 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:36.818683 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs"] Aug 05 18:31:36.822613 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:36.822589 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b5b093-fsfcs"] Aug 05 18:31:38.068977 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:38.068941 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84e6767-2c5f-49dc-a20b-92b1a43397a2" path="/var/lib/kubelet/pods/d84e6767-2c5f-49dc-a20b-92b1a43397a2/volumes" Aug 05 18:31:44.217812 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:31:44.217777 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:32:27.100917 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:27.100882 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:32:27.101436 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:27.101078 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" containerID="cri-o://258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc" gracePeriod=30 Aug 05 18:32:32.349542 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.349507 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:32:32.473024 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.472937 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473024 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.472978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473024 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473002 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473050 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473074 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473260 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473304 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473338 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473330 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473361 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnnc9\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9\") pod \"c642a85e-c872-4bff-9c90-8eb36eb52423\" (UID: \"c642a85e-c872-4bff-9c90-8eb36eb52423\") " Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473260 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data" (OuterVolumeSpecName: "istio-data") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473292 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473473 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473507 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:32.473581 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473518 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:32:32.473884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473688 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c642a85e-c872-4bff-9c90-8eb36eb52423-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.473884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473708 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.473884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473721 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.473884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473732 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.473884 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.473741 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.475593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.475559 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token" (OuterVolumeSpecName: "istio-token") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:32:32.475695 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.475621 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:32.475695 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.475637 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:32:32.475767 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.475727 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9" (OuterVolumeSpecName: "kube-api-access-cnnc9") pod "c642a85e-c872-4bff-9c90-8eb36eb52423" (UID: "c642a85e-c872-4bff-9c90-8eb36eb52423"). InnerVolumeSpecName "kube-api-access-cnnc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:32:32.574447 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.574399 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.574447 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.574440 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.574447 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.574450 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnc9\" (UniqueName: \"kubernetes.io/projected/c642a85e-c872-4bff-9c90-8eb36eb52423-kube-api-access-cnnc9\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.574447 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.574461 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c642a85e-c872-4bff-9c90-8eb36eb52423-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:32.814342 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.814311 2561 generic.go:356] "Generic (PLEG): container finished" podID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerID="258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc" exitCode=0 Aug 05 18:32:32.814514 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.814362 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" Aug 05 18:32:32.814514 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.814364 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerDied","Data":"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc"} Aug 05 18:32:32.814514 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.814491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch" event={"ID":"c642a85e-c872-4bff-9c90-8eb36eb52423","Type":"ContainerDied","Data":"63a2662452dc25bcbb4a4bcdf48e2c905367e0b6a0905dbaf5976d0e69a00a68"} Aug 05 18:32:32.814616 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.814525 2561 scope.go:117] "RemoveContainer" containerID="258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc" Aug 05 18:32:32.824328 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.824309 2561 scope.go:117] "RemoveContainer" containerID="79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235" Aug 05 18:32:32.832041 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.832023 2561 scope.go:117] "RemoveContainer" containerID="258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc" Aug 05 18:32:32.832318 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:32:32.832298 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc\": container with ID starting with 258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc not found: ID does not exist" containerID="258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc" Aug 05 18:32:32.832362 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.832329 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc"} err="failed to get container status \"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc\": rpc error: code = NotFound desc = could not find container \"258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc\": container with ID starting with 258814fc828548a561a13a5c12ab46e93f868f71702df69f56ad7b3cb72b0dfc not found: ID does not exist" Aug 05 18:32:32.832362 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.832345 2561 scope.go:117] "RemoveContainer" containerID="79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235" Aug 05 18:32:32.832580 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:32:32.832564 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235\": container with ID starting with 79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235 not found: ID does not exist" containerID="79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235" Aug 05 18:32:32.832618 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.832587 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235"} err="failed to get container status \"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235\": rpc error: code = NotFound desc = could not find container \"79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235\": container with ID starting with 79288080555dff302f227c0a0491cd80e6fc43c08c285b1b219ae590f3c4d235 not found: ID does not exist" Aug 05 18:32:32.839417 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.839393 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:32:32.842366 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:32.842344 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b5b093-openshift-default-788d79fc96-5wwch"] Aug 05 18:32:34.070061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:34.070028 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" path="/var/lib/kubelet/pods/c642a85e-c872-4bff-9c90-8eb36eb52423/volumes" Aug 05 18:32:40.403311 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:40.403280 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:32:45.409205 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:45.409131 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:32:45.409635 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:45.409390 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" containerID="cri-o://c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08" gracePeriod=30 Aug 05 18:32:50.664160 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.664137 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:32:50.729774 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729699 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.729774 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729737 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.729774 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729761 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm8fb\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729780 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729801 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729937 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729987 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:50.730061 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.729997 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730081 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730094 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:50.730356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730115 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy\") pod \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\" (UID: \"0458cf14-88a1-4bd9-b887-360ff6b3ecc2\") " Aug 05 18:32:50.730356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730110 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:50.730356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730320 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:50.730817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730386 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:32:50.730817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730442 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.730817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730459 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.730817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730471 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.730817 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.730481 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.732275 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.732203 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb" (OuterVolumeSpecName: "kube-api-access-rm8fb") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "kube-api-access-rm8fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:32:50.732275 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.732214 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:32:50.732407 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.732295 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:32:50.732444 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.732417 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0458cf14-88a1-4bd9-b887-360ff6b3ecc2" (UID: "0458cf14-88a1-4bd9-b887-360ff6b3ecc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:32:50.831013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.830978 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.831013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.831007 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.831013 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.831016 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.831253 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.831023 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.831253 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.831038 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm8fb\" (UniqueName: \"kubernetes.io/projected/0458cf14-88a1-4bd9-b887-360ff6b3ecc2-kube-api-access-rm8fb\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:32:50.893396 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.893362 2561 generic.go:356] "Generic (PLEG): container finished" podID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerID="c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08" exitCode=0 Aug 05 18:32:50.893538 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.893447 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" Aug 05 18:32:50.893538 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.893446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerDied","Data":"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08"} Aug 05 18:32:50.893538 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.893535 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf" event={"ID":"0458cf14-88a1-4bd9-b887-360ff6b3ecc2","Type":"ContainerDied","Data":"e5651d8d080a24a427e0b8c0ccd8f9f48cf2f863b19c61527ce4421bc21003f4"} Aug 05 18:32:50.893694 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.893560 2561 scope.go:117] "RemoveContainer" containerID="c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08" Aug 05 18:32:50.902680 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.902663 2561 scope.go:117] "RemoveContainer" containerID="5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75" Aug 05 18:32:50.910032 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.910014 2561 scope.go:117] "RemoveContainer" containerID="c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08" Aug 05 18:32:50.910291 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:32:50.910269 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08\": container with ID starting with c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08 not found: ID does not exist" containerID="c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08" Aug 05 18:32:50.910356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.910297 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08"} err="failed to get container status \"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08\": rpc error: code = NotFound desc = could not find container \"c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08\": container with ID starting with c73c4a1c3209612ad2c08b2546a960cb6267916e8ca813a7b2b6dc746a193f08 not found: ID does not exist" Aug 05 18:32:50.910356 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.910316 2561 scope.go:117] "RemoveContainer" containerID="5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75" Aug 05 18:32:50.910552 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:32:50.910536 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75\": container with ID starting with 5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75 not found: ID does not exist" containerID="5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75" Aug 05 18:32:50.910592 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.910558 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75"} err="failed to get container status \"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75\": rpc error: code = NotFound desc = could not find container \"5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75\": container with ID starting with 5e13b0266ea9c6a8fc1a0858aab256a7b03be6edb682bfd21110188cf9c4af75 not found: ID does not exist" Aug 05 18:32:50.916417 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.916395 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:32:50.922997 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:50.922976 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cea6f9-openshift-default-75b9b94cdd-56xkf"] Aug 05 18:32:52.070188 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:52.070154 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" path="/var/lib/kubelet/pods/0458cf14-88a1-4bd9-b887-360ff6b3ecc2/volumes" Aug 05 18:32:54.199933 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:54.199909 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:32:54.203791 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:54.203767 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:32:57.908127 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:32:57.908091 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:33:03.896907 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:03.896875 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:33:03.897485 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:03.897087 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" containerID="cri-o://2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e" gracePeriod=30 Aug 05 18:33:09.142486 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.142463 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:33:09.188251 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188172 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188251 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188204 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjsfv\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188254 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188292 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188424 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188345 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188424 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188480 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188588 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188504 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data\") pod \"a17e22fd-2d7c-4063-a755-c53ff58685cc\" (UID: \"a17e22fd-2d7c-4063-a755-c53ff58685cc\") " Aug 05 18:33:09.188756 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188715 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:09.188947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188813 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:33:09.188947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188898 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:09.188947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188913 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:09.189133 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.188969 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:09.190625 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.190602 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:33:09.190982 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.190954 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:33:09.191111 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.191092 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:09.191156 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.191109 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv" (OuterVolumeSpecName: "kube-api-access-vjsfv") pod "a17e22fd-2d7c-4063-a755-c53ff58685cc" (UID: "a17e22fd-2d7c-4063-a755-c53ff58685cc"). InnerVolumeSpecName "kube-api-access-vjsfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:33:09.289284 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289215 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289284 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289279 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289284 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289289 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17e22fd-2d7c-4063-a755-c53ff58685cc-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289284 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289298 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289306 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289314 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289321 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17e22fd-2d7c-4063-a755-c53ff58685cc-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289329 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsfv\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-kube-api-access-vjsfv\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.289544 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.289340 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17e22fd-2d7c-4063-a755-c53ff58685cc-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:09.975172 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.975139 2561 generic.go:356] "Generic (PLEG): container finished" podID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerID="2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e" exitCode=0 Aug 05 18:33:09.975373 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.975194 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" Aug 05 18:33:09.975373 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.975237 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerDied","Data":"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e"} Aug 05 18:33:09.975373 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.975315 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg" event={"ID":"a17e22fd-2d7c-4063-a755-c53ff58685cc","Type":"ContainerDied","Data":"9649891b48c7dee2c052cedb6e7b7801b4beea4958a284340a538a63d7b15392"} Aug 05 18:33:09.975373 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.975339 2561 scope.go:117] "RemoveContainer" containerID="2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e" Aug 05 18:33:09.983947 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.983930 2561 scope.go:117] "RemoveContainer" containerID="dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6" Aug 05 18:33:09.991554 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.991530 2561 scope.go:117] "RemoveContainer" containerID="2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e" Aug 05 18:33:09.991804 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:33:09.991785 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e\": container with ID starting with 2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e not found: ID does not exist" containerID="2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e" Aug 05 18:33:09.991880 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.991816 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e"} err="failed to get container status \"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e\": rpc error: code = NotFound desc = could not find container \"2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e\": container with ID starting with 2691475878f618d8fdb170e2ca045928dd04a33cf120296bc47f27ed19cfa87e not found: ID does not exist" Aug 05 18:33:09.991880 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.991839 2561 scope.go:117] "RemoveContainer" containerID="dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6" Aug 05 18:33:09.992097 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:33:09.992081 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6\": container with ID starting with dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6 not found: ID does not exist" containerID="dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6" Aug 05 18:33:09.992137 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.992103 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6"} err="failed to get container status \"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6\": rpc error: code = NotFound desc = could not find container \"dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6\": container with ID starting with dc3947c9bcf9d60d23de17a82353d50f9c1f72179200144725355d816a706cb6 not found: ID does not exist" Aug 05 18:33:09.996636 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:09.996608 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:33:10.004095 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:10.004074 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-70b4a4-openshift-default-b8f48db56-l5wqg"] Aug 05 18:33:10.068370 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:10.068345 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" path="/var/lib/kubelet/pods/a17e22fd-2d7c-4063-a755-c53ff58685cc/volumes" Aug 05 18:33:17.319220 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:17.319188 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hcw6v"] Aug 05 18:33:22.487820 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:22.487789 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:33:22.488219 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:22.487987 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" containerID="cri-o://af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71" gracePeriod=30 Aug 05 18:33:27.728606 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.728579 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:33:27.836507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836431 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836507 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836482 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836512 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836539 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836571 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4446\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836610 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836650 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836698 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.836731 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836725 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs\") pod \"d9e59029-a89f-46df-8a54-58478a07b0f7\" (UID: \"d9e59029-a89f-46df-8a54-58478a07b0f7\") " Aug 05 18:33:27.837077 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836856 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:27.837077 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.836946 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 05 18:33:27.837077 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.837040 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-credential-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.837213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.837093 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:27.837213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.837194 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:27.837213 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.837200 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:27.838785 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.838757 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:33:27.839104 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.839082 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 05 18:33:27.839361 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.839333 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 05 18:33:27.839454 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.839351 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446" (OuterVolumeSpecName: "kube-api-access-j4446") pod "d9e59029-a89f-46df-8a54-58478a07b0f7" (UID: "d9e59029-a89f-46df-8a54-58478a07b0f7"). InnerVolumeSpecName "kube-api-access-j4446". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 05 18:33:27.938157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938115 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4446\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-kube-api-access-j4446\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938155 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-envoy\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938165 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-socket\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938157 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938173 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-data\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938181 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e59029-a89f-46df-8a54-58478a07b0f7-workload-certs\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938188 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-token\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938196 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e59029-a89f-46df-8a54-58478a07b0f7-istio-podinfo\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:27.938421 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:27.938203 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e59029-a89f-46df-8a54-58478a07b0f7-istiod-ca-cert\") on node \"ip-10-0-143-87.ec2.internal\" DevicePath \"\"" Aug 05 18:33:28.057053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.057017 2561 generic.go:356] "Generic (PLEG): container finished" podID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerID="af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71" exitCode=0 Aug 05 18:33:28.057211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.057066 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" Aug 05 18:33:28.057211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.057080 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerDied","Data":"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71"} Aug 05 18:33:28.057211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.057132 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4" event={"ID":"d9e59029-a89f-46df-8a54-58478a07b0f7","Type":"ContainerDied","Data":"9a627bedada211b7c0d48c73b0040adf6218f5b9116302fae279a8c2ff7ebe20"} Aug 05 18:33:28.057211 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.057158 2561 scope.go:117] "RemoveContainer" containerID="af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71" Aug 05 18:33:28.066970 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.066950 2561 scope.go:117] "RemoveContainer" containerID="88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e" Aug 05 18:33:28.075121 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.075098 2561 scope.go:117] "RemoveContainer" containerID="af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71" Aug 05 18:33:28.075405 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:33:28.075386 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71\": container with ID starting with af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71 not found: ID does not exist" containerID="af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71" Aug 05 18:33:28.075477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.075415 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71"} err="failed to get container status \"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71\": rpc error: code = NotFound desc = could not find container \"af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71\": container with ID starting with af1a689429bb299a7762fd84391c89d8eacf092b760cf7ac793161b880555e71 not found: ID does not exist" Aug 05 18:33:28.075477 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.075433 2561 scope.go:117] "RemoveContainer" containerID="88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e" Aug 05 18:33:28.075682 ip-10-0-143-87 kubenswrapper[2561]: E0805 18:33:28.075664 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e\": container with ID starting with 88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e not found: ID does not exist" containerID="88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e" Aug 05 18:33:28.075728 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.075690 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e"} err="failed to get container status \"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e\": rpc error: code = NotFound desc = could not find container \"88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e\": container with ID starting with 88165f494994af8db2060c4fcf21c8c0a4954ac648e05067ffb748def9e9a31e not found: ID does not exist" Aug 05 18:33:28.082057 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.082036 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:33:28.086404 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:28.086381 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-6fb560-openshift-default-64c67f8b84-94gj4"] Aug 05 18:33:29.325003 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:29.324976 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856dccdfcf-6cn8z_d505ec41-5568-415d-b0f2-ba6fbd311339/copy-manifests/0.log" Aug 05 18:33:29.331799 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:29.331778 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856dccdfcf-6cn8z_d505ec41-5568-415d-b0f2-ba6fbd311339/manager/0.log" Aug 05 18:33:29.913030 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:29.913002 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29765880-5c6zs_a78af49b-0cda-4735-8755-0a3282baaff6/cleanup/0.log" Aug 05 18:33:30.022347 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:30.022314 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29765895-nqzcp_ccca35ad-eb6a-4b8d-9953-34bebb261290/cleanup/0.log" Aug 05 18:33:30.068530 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:30.068500 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" path="/var/lib/kubelet/pods/d9e59029-a89f-46df-8a54-58478a07b0f7/volumes" Aug 05 18:33:31.555473 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:31.555448 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-hcw6v_ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6/limitador/0.log" Aug 05 18:33:32.135896 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:32.135870 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lpf2k_c02c55d1-d6d4-4ae2-9427-52c32f294adb/discovery/0.log" Aug 05 18:33:32.833263 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:32.833219 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86984d778b-h5jmk_f7963633-3133-4b6d-af46-d0dd588a423c/router/0.log" Aug 05 18:33:33.405931 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:33.405904 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-c8kdr_8e0007cc-fca2-40bd-bfad-6c76833b8582/main/0.log" Aug 05 18:33:41.608107 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:41.608077 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4mfcd_18f36257-69b8-428e-b68b-4913f2dad3cf/global-pull-secret-syncer/0.log" Aug 05 18:33:41.747383 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:41.747356 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6hk6k_88736750-2f0f-4a39-a0db-075d627e9b1d/konnectivity-agent/0.log" Aug 05 18:33:41.875583 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:41.875501 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-87.ec2.internal_6f01ba913ced227a9afb502c83707f6b/haproxy/0.log" Aug 05 18:33:47.482243 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:47.482204 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-hcw6v_ac4cccd0-356c-409f-acf0-ff9ef1a0d6c6/limitador/0.log" Aug 05 18:33:49.494098 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:49.494072 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkl6c_5c83618d-c200-409b-ba0d-9bb3b332a4ef/node-exporter/0.log" Aug 05 18:33:49.524631 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:49.524597 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkl6c_5c83618d-c200-409b-ba0d-9bb3b332a4ef/kube-rbac-proxy/0.log" Aug 05 18:33:49.548726 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:49.548704 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hkl6c_5c83618d-c200-409b-ba0d-9bb3b332a4ef/init-textfile/0.log" Aug 05 18:33:51.473593 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473559 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh"] Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473835 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473846 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473855 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473860 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473871 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84e6767-2c5f-49dc-a20b-92b1a43397a2" containerName="revoke-keys" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473876 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84e6767-2c5f-49dc-a20b-92b1a43397a2" containerName="revoke-keys" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473883 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473888 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473897 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473902 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473908 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473914 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473919 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473924 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473931 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473936 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473946 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" Aug 05 18:33:51.473964 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473951 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.473996 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e59029-a89f-46df-8a54-58478a07b0f7" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474003 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84e6767-2c5f-49dc-a20b-92b1a43397a2" containerName="revoke-keys" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474009 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474016 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474020 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0458cf14-88a1-4bd9-b887-360ff6b3ecc2" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474026 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c642a85e-c872-4bff-9c90-8eb36eb52423" containerName="istio-proxy" Aug 05 18:33:51.474546 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.474030 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17e22fd-2d7c-4063-a755-c53ff58685cc" containerName="istio-proxy" Aug 05 18:33:51.476824 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.476805 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.479350 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.479331 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rl9xm\"/\"kube-root-ca.crt\"" Aug 05 18:33:51.480155 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.480139 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rl9xm\"/\"openshift-service-ca.crt\"" Aug 05 18:33:51.480260 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.480179 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rl9xm\"/\"default-dockercfg-62k6z\"" Aug 05 18:33:51.485705 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.485685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh"] Aug 05 18:33:51.628561 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.628523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-podres\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.628768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.628574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-proc\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.628768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.628627 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-lib-modules\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.628768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.628674 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-sys\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.628768 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.628697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpnj\" (UniqueName: \"kubernetes.io/projected/62013f52-df30-45b7-9718-809acb860211-kube-api-access-2zpnj\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.729905 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729815 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-proc\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.729905 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-lib-modules\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.729905 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729885 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-sys\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.729905 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpnj\" (UniqueName: \"kubernetes.io/projected/62013f52-df30-45b7-9718-809acb860211-kube-api-access-2zpnj\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.730159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729937 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-podres\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.730159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729903 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-proc\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.730159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.729998 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-lib-modules\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.730159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.730028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-sys\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.730159 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.730060 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/62013f52-df30-45b7-9718-809acb860211-podres\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.739641 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.739617 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpnj\" (UniqueName: \"kubernetes.io/projected/62013f52-df30-45b7-9718-809acb860211-kube-api-access-2zpnj\") pod \"perf-node-gather-daemonset-fwjmh\" (UID: \"62013f52-df30-45b7-9718-809acb860211\") " pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.786529 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.786506 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:51.917915 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:51.917890 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh"] Aug 05 18:33:51.919891 ip-10-0-143-87 kubenswrapper[2561]: W0805 18:33:51.919860 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod62013f52_df30_45b7_9718_809acb860211.slice/crio-e2795811d53ad353516c1861e8831e16c3e8306ca5affbe135a862387651e239 WatchSource:0}: Error finding container e2795811d53ad353516c1861e8831e16c3e8306ca5affbe135a862387651e239: Status 404 returned error can't find the container with id e2795811d53ad353516c1861e8831e16c3e8306ca5affbe135a862387651e239 Aug 05 18:33:52.155194 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.155163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" event={"ID":"62013f52-df30-45b7-9718-809acb860211","Type":"ContainerStarted","Data":"698792a0d97b85a2b2b01e56f4d9d05c3f0af9651d656b47c88c23eed6b3c1d0"} Aug 05 18:33:52.155402 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.155219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" event={"ID":"62013f52-df30-45b7-9718-809acb860211","Type":"ContainerStarted","Data":"e2795811d53ad353516c1861e8831e16c3e8306ca5affbe135a862387651e239"} Aug 05 18:33:52.155402 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.155274 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:52.178942 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.178887 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" podStartSLOduration=1.1788670940000001 podStartE2EDuration="1.178867094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 18:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:33:52.177798133 +0000 UTC m=+3658.654080597" watchObservedRunningTime="2026-08-05 18:33:52.178867094 +0000 UTC m=+3658.655149560" Aug 05 18:33:52.269702 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.269678 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/1.log" Aug 05 18:33:52.275336 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:52.275312 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-nrs7p_1be440ce-b9fe-486b-8edd-5fc930de9c3c/console-operator/2.log" Aug 05 18:33:54.818743 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:54.818713 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nxfjd_7de59047-10b9-441d-814d-2654e9f013a9/dns/0.log" Aug 05 18:33:54.842161 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:54.842136 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nxfjd_7de59047-10b9-441d-814d-2654e9f013a9/kube-rbac-proxy/0.log" Aug 05 18:33:54.923150 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:54.923125 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w965k_1382e3a7-4675-4b1c-a5a3-a193ab8f54a5/dns-node-resolver/0.log" Aug 05 18:33:55.644740 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:55.644712 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5t2w_e2436a4d-8c40-44f6-84c2-3c7c8663a80c/node-ca/0.log" Aug 05 18:33:56.924209 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:56.924177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lpf2k_c02c55d1-d6d4-4ae2-9427-52c32f294adb/discovery/0.log" Aug 05 18:33:57.135100 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:57.135069 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86984d778b-h5jmk_f7963633-3133-4b6d-af46-d0dd588a423c/router/0.log" Aug 05 18:33:58.068214 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:58.068185 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nntms_817f6abd-e663-4560-b4d8-a7eaf8697446/serve-healthcheck-canary/0.log" Aug 05 18:33:58.169398 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:58.169364 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rl9xm/perf-node-gather-daemonset-fwjmh" Aug 05 18:33:58.655220 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:58.655191 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5cw_1207f1cd-50a0-4bbe-a218-004804dd5506/kube-rbac-proxy/0.log" Aug 05 18:33:58.676675 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:58.676653 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5cw_1207f1cd-50a0-4bbe-a218-004804dd5506/exporter/0.log" Aug 05 18:33:58.698053 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:33:58.698033 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rf5cw_1207f1cd-50a0-4bbe-a218-004804dd5506/extractor/0.log" Aug 05 18:34:02.376753 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:02.376725 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lpf2k_c02c55d1-d6d4-4ae2-9427-52c32f294adb/discovery/0.log" Aug 05 18:34:02.550116 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:02.550087 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86984d778b-h5jmk_f7963633-3133-4b6d-af46-d0dd588a423c/router/0.log" Aug 05 18:34:02.625563 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:02.625475 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856dccdfcf-6cn8z_d505ec41-5568-415d-b0f2-ba6fbd311339/manager/0.log" Aug 05 18:34:02.663025 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:02.663002 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856dccdfcf-6cn8z_d505ec41-5568-415d-b0f2-ba6fbd311339/copy-manifests/0.log" Aug 05 18:34:04.039393 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:04.039361 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-dd9d7cdcd-cpxnq_d63a9e85-c233-4dcb-ab8f-91196f38a0f2/manager/0.log" Aug 05 18:34:04.186177 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:04.186147 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lpf2k_c02c55d1-d6d4-4ae2-9427-52c32f294adb/discovery/0.log" Aug 05 18:34:04.369815 ip-10-0-143-87 kubenswrapper[2561]: I0805 18:34:04.369738 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86984d778b-h5jmk_f7963633-3133-4b6d-af46-d0dd588a423c/router/0.log"